Compare commits

..

1 Commits
v0.12.0 ... vcr

Author SHA1 Message Date
ed
ff8313d0fb add mistake 2021-07-01 21:49:44 +02:00
54 changed files with 1722 additions and 3903 deletions

5
.vscode/tasks.json vendored
View File

@@ -9,10 +9,7 @@
{ {
"label": "no_dbg", "label": "no_dbg",
"type": "shell", "type": "shell",
"command": "${config:python.pythonPath}", "command": "${config:python.pythonPath} .vscode/launch.py"
"args": [
".vscode/launch.py"
]
} }
] ]
} }

225
README.md
View File

@@ -16,11 +16,6 @@ turn your phone or raspi into a portable file server with resumable uploads/down
📷 **screenshots:** [browser](#the-browser) // [upload](#uploading) // [thumbnails](#thumbnails) // [md-viewer](#markdown-viewer) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [ie4](#browser-support) 📷 **screenshots:** [browser](#the-browser) // [upload](#uploading) // [thumbnails](#thumbnails) // [md-viewer](#markdown-viewer) // [search](#searching) // [fsearch](#file-search) // [zip-DL](#zip-downloads) // [ie4](#browser-support)
## breaking changes \o/
this is the readme for v0.12 which has a different expression for volume permissions (`-v`); see [the v0.11.x readme](https://github.com/9001/copyparty/tree/15b59822112dda56cee576df30f331252fc62628#readme) for stuff regarding the [current stable release](https://github.com/9001/copyparty/releases/tag/v0.11.47)
## readme toc ## readme toc
* top * top
@@ -28,19 +23,17 @@ this is the readme for v0.12 which has a different expression for volume permiss
* [on debian](#on-debian) * [on debian](#on-debian)
* [notes](#notes) * [notes](#notes)
* [status](#status) * [status](#status)
* [testimonials](#testimonials)
* [bugs](#bugs) * [bugs](#bugs)
* [general bugs](#general-bugs) * [general bugs](#general-bugs)
* [not my bugs](#not-my-bugs) * [not my bugs](#not-my-bugs)
* [the browser](#the-browser) * [the browser](#the-browser)
* [tabs](#tabs) * [tabs](#tabs)
* [hotkeys](#hotkeys) * [hotkeys](#hotkeys)
* [navpane](#navpane) * [tree-mode](#tree-mode)
* [thumbnails](#thumbnails) * [thumbnails](#thumbnails)
* [zip downloads](#zip-downloads) * [zip downloads](#zip-downloads)
* [uploading](#uploading) * [uploading](#uploading)
* [file-search](#file-search) * [file-search](#file-search)
* [file manager](#file-manager)
* [markdown viewer](#markdown-viewer) * [markdown viewer](#markdown-viewer)
* [other tricks](#other-tricks) * [other tricks](#other-tricks)
* [searching](#searching) * [searching](#searching)
@@ -52,7 +45,6 @@ this is the readme for v0.12 which has a different expression for volume permiss
* [browser support](#browser-support) * [browser support](#browser-support)
* [client examples](#client-examples) * [client examples](#client-examples)
* [up2k](#up2k) * [up2k](#up2k)
* [performance](#performance)
* [dependencies](#dependencies) * [dependencies](#dependencies)
* [optional dependencies](#optional-dependencies) * [optional dependencies](#optional-dependencies)
* [install recommended deps](#install-recommended-deps) * [install recommended deps](#install-recommended-deps)
@@ -65,21 +57,21 @@ this is the readme for v0.12 which has a different expression for volume permiss
* [just the sfx](#just-the-sfx) * [just the sfx](#just-the-sfx)
* [complete release](#complete-release) * [complete release](#complete-release)
* [todo](#todo) * [todo](#todo)
* [discarded ideas](#discarded-ideas)
## quickstart ## quickstart
download [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) and you're all set! download [copyparty-sfx.py](https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py) and you're all set!
running the sfx without arguments (for example doubleclicking it on Windows) will give everyone full access to the current folder; see `-h` for help if you want [accounts and volumes](#accounts-and-volumes) etc running the sfx without arguments (for example doubleclicking it on Windows) will give everyone full access to the current folder; see `-h` for help if you want accounts and volumes etc
some recommended options: some recommended options:
* `-e2dsa` enables general file indexing, see [search configuration](#search-configuration) * `-e2dsa` enables general file indexing, see [search configuration](#search-configuration)
* `-e2ts` enables audio metadata indexing (needs either FFprobe or Mutagen), see [optional dependencies](#optional-dependencies) * `-e2ts` enables audio metadata indexing (needs either FFprobe or mutagen), see [optional dependencies](#optional-dependencies)
* `-v /mnt/music:/music:r:rw,foo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, and read-write for user `foo`, password `bar` * `-v /mnt/music:/music:r:afoo -a foo:bar` shares `/mnt/music` as `/music`, `r`eadable by anyone, with user `foo` as `a`dmin (read/write), password `bar`
* replace `:r:rw,foo` with `:r,foo` to only make the folder readable by `foo` and nobody else * the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
* see [accounts and volumes](#accounts-and-volumes) for the syntax and other access levels (`r`ead, `w`rite, `m`ove, `d`elete) * replace `:r:afoo` with `:rfoo` to only make the folder readable by `foo` and nobody else
* in addition to `r`ead and `a`dmin, `w`rite makes a folder write-only, so cannot list/access files in it
* `--ls '**,*,ln,p,r'` to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access * `--ls '**,*,ln,p,r'` to crash on startup if any of the volumes contain a symlink which point outside the volume, as that could give users unintended access
you may also want these, especially on servers: you may also want these, especially on servers:
@@ -120,9 +112,9 @@ summary: all planned features work! now please enjoy the bloatening
* backend stuff * backend stuff
* ☑ sanic multipart parser * ☑ sanic multipart parser
*multiprocessing (actual multithreading) *load balancer (multiprocessing)
* ☑ volumes (mountpoints) * ☑ volumes (mountpoints)
*[accounts](#accounts-and-volumes) * ☑ accounts
* upload * upload
* ☑ basic: plain multipart, ie6 support * ☑ basic: plain multipart, ie6 support
* ☑ up2k: js, resumable, multithreaded * ☑ up2k: js, resumable, multithreaded
@@ -133,15 +125,15 @@ summary: all planned features work! now please enjoy the bloatening
* ☑ folders as zip / tar files * ☑ folders as zip / tar files
* ☑ FUSE client (read-only) * ☑ FUSE client (read-only)
* browser * browser
*navpane (directory tree sidebar) *tree-view
* ☑ audio player (with OS media controls) * ☑ audio player (with OS media controls)
* ☑ thumbnails * ☑ thumbnails
* ...of images using Pillow * ☑ images using Pillow
* ...of videos using FFmpeg * ☑ videos using FFmpeg
* ☑ cache eviction (max-age; maybe max-size eventually) * ☑ cache eviction (max-age; maybe max-size eventually)
* ☑ image gallery with webm player * ☑ image gallery
* ☑ SPA (browse while uploading) * ☑ SPA (browse while uploading)
* if you use the navpane to navigate, not folders in the file list * if you use the file-tree on the left only, not folders in the file list
* server indexing * server indexing
* ☑ locate files by contents * ☑ locate files by contents
* ☑ search by name/path/date/size * ☑ search by name/path/date/size
@@ -151,24 +143,19 @@ summary: all planned features work! now please enjoy the bloatening
* ☑ editor (sure why not) * ☑ editor (sure why not)
## testimonials
small collection of user feedback
`good enough`, `surprisingly correct`, `certified good software`, `just works`, `why`
# bugs # bugs
* Windows: python 3.7 and older cannot read tags with FFprobe, so use Mutagen or upgrade * Windows: python 3.7 and older cannot read tags with ffprobe, so use mutagen or upgrade
* Windows: python 2.7 cannot index non-ascii filenames with `-e2d` * Windows: python 2.7 cannot index non-ascii filenames with `-e2d`
* Windows: python 2.7 cannot handle filenames with mojibake * Windows: python 2.7 cannot handle filenames with mojibake
* `--th-ff-jpg` may fix video thumbnails on some FFmpeg versions * MacOS: `--th-ff-jpg` may fix thumbnails using macports-FFmpeg
## general bugs ## general bugs
* all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise * all volumes must exist / be available on startup; up2k (mtp especially) gets funky otherwise
* cannot mount something at `/d1/d2/d3` unless `d2` exists inside `d1` * cannot mount something at `/d1/d2/d3` unless `d2` exists inside `d1`
* dupe files will not have metadata (audio tags etc) displayed in the file listing
* because they don't get `up` entries in the db (probably best fix) and `tx_browser` does not `lstat`
* probably more, pls let me know * probably more, pls let me know
## not my bugs ## not my bugs
@@ -176,36 +163,9 @@ small collection of user feedback
* Windows: folders cannot be accessed if the name ends with `.` * Windows: folders cannot be accessed if the name ends with `.`
* python or windows bug * python or windows bug
* Windows: msys2-python 3.8.6 occasionally throws `RuntimeError: release unlocked lock` when leaving a scoped mutex in up2k * Windows: msys2-python 3.8.6 occasionally throws "RuntimeError: release unlocked lock" when leaving a scoped mutex in up2k
* this is an msys2 bug, the regular windows edition of python is fine * this is an msys2 bug, the regular windows edition of python is fine
* VirtualBox: sqlite throws `Disk I/O Error` when running in a VM and the up2k database is in a vboxsf
* use `--hist` or the `hist` volflag (`-v [...]:chist=/tmp/foo`) to place the db inside the vm instead
# accounts and volumes
* `-a usr:pwd` adds account `usr` with password `pwd`
* `-v .::r` adds current-folder `.` as the webroot, `r`eadable by anyone
* the syntax is `-v src:dst:perm:perm:...` so local-path, url-path, and one or more permissions to set
* when granting permissions to an account, the names are comma-separated: `-v .::r,usr1,usr2:rw,usr3,usr4`
permissions:
* `r` (read): browse folder contents, download files, download as zip/tar
* `w` (write): upload files, move files *into* folder
* `m` (move): move files/folders *from* folder
* `d` (delete): delete files/folders
example:
* add accounts named u1, u2, u3 with passwords p1, p2, p3: `-a u1:p1 -a u2:p2 -a u3:p3`
* make folder `/srv` the root of the filesystem, read-only by anyone: `-v /srv::r`
* make folder `/mnt/music` available at `/music`, read-only for u1 and u2, read-write for u3: `-v /mnt/music:music:r,u1,u2:rw,u3`
* unauthorized users accessing the webroot can see that the `music` folder exists, but cannot open it
* make folder `/mnt/incoming` available at `/inc`, write-only for u1, read-move for u2: `-v /mnt/incoming:inc:w,u1:rm,u2`
* unauthorized users accessing the webroot can see that the `inc` folder exists, but cannot open it
* `u1` can open the `inc` folder, but cannot see the contents, only upload new files to it
* `u2` can browse it and move files *from* `/inc` into any folder where `u2` has write-access
# the browser # the browser
@@ -219,60 +179,32 @@ example:
* `[📂]` mkdir, create directories * `[📂]` mkdir, create directories
* `[📝]` new-md, create a new markdown document * `[📝]` new-md, create a new markdown document
* `[📟]` send-msg, either to server-log or into textfiles if `--urlform save` * `[📟]` send-msg, either to server-log or into textfiles if `--urlform save`
* `[🎺]` audio-player config options * `[⚙️]` client configuration options
* `[⚙️]` general client config options
## hotkeys ## hotkeys
the browser has the following hotkeys (assumes qwerty, ignores actual layout) the browser has the following hotkeys
* `B` toggle breadcrumbs / navpane * `B` toggle breadcrumbs / directory tree
* `I/K` prev/next folder * `I/K` prev/next folder
* `M` parent folder (or unexpand current) * `M` parent folder
* `G` toggle list / grid view * `G` toggle list / grid view
* `T` toggle thumbnails / icons * `T` toggle thumbnails / icons
* `ctrl-X` cut selected files/folders
* `ctrl-V` paste
* `F2` rename selected file/folder
* when a file/folder is selected (in not-grid-view):
* `Up/Down` move cursor
* shift+`Up/Down` select and move cursor
* ctrl+`Up/Down` move cursor and scroll viewport
* `Space` toggle file selection
* `Ctrl-A` toggle select all
* when playing audio: * when playing audio:
* `J/L` prev/next song * `0..9` jump to 10%..90%
* `U/O` skip 10sec back/forward * `U/O` skip 10sec back/forward
* `0..9` jump to 0%..90% * `J/L` prev/next song
* `P` play/pause (also starts playing the folder) * `P` play/pause (also starts playing the folder)
* when viewing images / playing videos: * when tree-sidebar is open:
* `J/L, Left/Right` prev/next file
* `Home/End` first/last file
* `Esc` close viewer
* videos:
* `U/O` skip 10sec back/forward
* `P/K/Space` play/pause
* `F` fullscreen
* `C` continue playing next video
* `R` loop
* `M` mute
* when the navpane is open:
* `A/D` adjust tree width * `A/D` adjust tree width
* in the grid view: * in the grid view:
* `S` toggle multiselect * `S` toggle multiselect
* shift+`A/D` zoom * shift+`A/D` zoom
* in the markdown editor:
* `^s` save
* `^h` header
* `^k` autoformat table
* `^u` jump to next unicode character
* `^e` toggle editor / preview
* `^up, ^down` jump paragraphs
## navpane ## tree-mode
by default there's a breadcrumbs path; you can replace this with a navpane (tree-browser sidebar thing) by clicking the `🌲` or pressing the `B` hotkey by default there's a breadcrumbs path; you can replace this with a tree-browser sidebar thing by clicking the `🌲` or pressing the `B` hotkey
click `[-]` and `[+]` (or hotkeys `A`/`D`) to adjust the size, and the `[a]` toggles if the tree should widen dynamically as you go deeper or stay fixed-size click `[-]` and `[+]` (or hotkeys `A`/`D`) to adjust the size, and the `[a]` toggles if the tree should widen dynamically as you go deeper or stay fixed-size
@@ -281,9 +213,9 @@ click `[-]` and `[+]` (or hotkeys `A`/`D`) to adjust the size, and the `[a]` tog
![copyparty-thumbs-fs8](https://user-images.githubusercontent.com/241032/120070302-10836b00-c08a-11eb-8eb4-82004a34c342.png) ![copyparty-thumbs-fs8](https://user-images.githubusercontent.com/241032/120070302-10836b00-c08a-11eb-8eb4-82004a34c342.png)
it does static images with Pillow and uses FFmpeg for video files, so you may want to `--no-thumb` or maybe just `--no-vthumb` depending on how dangerous your users are it does static images with Pillow and uses FFmpeg for video files, so you may want to `--no-thumb` or maybe just `--no-vthumb` depending on how destructive your users are
images with the following names (see `--th-covers`) become the thumbnail of the folder they're in: `folder.png`, `folder.jpg`, `cover.png`, `cover.jpg` images named `folder.jpg` and `folder.png` become the thumbnail of the folder they're in
in the grid/thumbnail view, if the audio player panel is open, songs will start playing when clicked in the grid/thumbnail view, if the audio player panel is open, songs will start playing when clicked
@@ -300,10 +232,9 @@ the `zip` link next to folders can produce various types of zip/tar files using
| `zip_crc` | `?zip=crc` | cp437 with crc32 computed early for truly ancient software | | `zip_crc` | `?zip=crc` | cp437 with crc32 computed early for truly ancient software |
* hidden files (dotfiles) are excluded unless `-ed` * hidden files (dotfiles) are excluded unless `-ed`
* `up2k.db` and `dir.txt` is always excluded * the up2k.db is always excluded
* `zip_crc` will take longer to download since the server has to read each file twice * `zip_crc` will take longer to download since the server has to read each file twice
* this is only to support MS-DOS PKZIP v2.04g (october 1993) and older * please let me know if you find a program old enough to actually need this
* how are you accessing copyparty actually
you can also zip a selection of files or folders by clicking them in the browser, that brings up a selection editor and zip button in the bottom right you can also zip a selection of files or folders by clicking them in the browser, that brings up a selection editor and zip button in the bottom right
@@ -318,11 +249,9 @@ two upload methods are available in the html client:
up2k has several advantages: up2k has several advantages:
* you can drop folders into the browser (files are added recursively) * you can drop folders into the browser (files are added recursively)
* files are processed in chunks, and each chunk is checksummed * files are processed in chunks, and each chunk is checksummed
* uploads autoresume if they are interrupted by network issues * uploads resume if they are interrupted (for example by a reboot)
* uploads resume if you reboot your browser or pc, just upload the same files again
* server detects any corruption; the client reuploads affected chunks * server detects any corruption; the client reuploads affected chunks
* the client doesn't upload anything that already exists on the server * the client doesn't upload anything that already exists on the server
* much higher speeds than ftp/scp/tarpipe on some internet connections (mainly american ones) thanks to parallel connections
* the last-modified timestamp of the file is preserved * the last-modified timestamp of the file is preserved
see [up2k](#up2k) for details on how it works see [up2k](#up2k) for details on how it works
@@ -355,18 +284,11 @@ in the `[🚀 up2k]` tab, after toggling the `[🔎]` switch green, any files/fo
files go into `[ok]` if they exist (and you get a link to where it is), otherwise they land in `[ng]` files go into `[ok]` if they exist (and you get a link to where it is), otherwise they land in `[ng]`
* the main reason filesearch is combined with the uploader is cause the code was too spaghetti to separate it out somewhere else, this is no longer the case but now i've warmed up to the idea too much * the main reason filesearch is combined with the uploader is cause the code was too spaghetti to separate it out somewhere else, this is no longer the case but now i've warmed up to the idea too much
adding the same file multiple times is blocked, so if you first search for a file and then decide to upload it, you have to click the `[cleanup]` button to discard `[done]` files (or just refresh the page) adding the same file multiple times is blocked, so if you first search for a file and then decide to upload it, you have to click the `[cleanup]` button to discard `[done]` files
note that since up2k has to read the file twice, `[🎈 bup]` can be up to 2x faster in extreme cases (if your internet connection is faster than the read-speed of your HDD) note that since up2k has to read the file twice, `[🎈 bup]` can be up to 2x faster in extreme cases (if your internet connection is faster than the read-speed of your HDD)
up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well (thanks to tls also functioning as an integrity check) up2k has saved a few uploads from becoming corrupted in-transfer already; caught an android phone on wifi redhanded in wireshark with a bitflip, however bup with https would *probably* have noticed as well thanks to tls also functioning as an integrity check
## file manager
if you have the required permissions, you can cut/paste, rename, and delete files/folders
you can move files across browser tabs (cut in one tab, paste in another)
## markdown viewer ## markdown viewer
@@ -404,11 +326,11 @@ searching relies on two databases, the up2k filetree (`-e2d`) and the metadata t
through arguments: through arguments:
* `-e2d` enables file indexing on upload * `-e2d` enables file indexing on upload
* `-e2ds` scans writable folders for new files on startup * `-e2ds` scans writable folders on startup
* `-e2dsa` scans all mounted volumes (including readonly ones) * `-e2dsa` scans all mounted volumes (including readonly ones)
* `-e2t` enables metadata indexing on upload * `-e2t` enables metadata indexing on upload
* `-e2ts` scans for tags in all files that don't have tags yet * `-e2ts` scans for tags in all files that don't have tags yet
* `-e2tsr` deletes all existing tags, does a full reindex * `-e2tsr` deletes all existing tags, so a full reindex
the same arguments can be set as volume flags, in addition to `d2d` and `d2t` for disabling: the same arguments can be set as volume flags, in addition to `d2d` and `d2t` for disabling:
* `-v ~/music::r:ce2dsa:ce2tsr` does a full reindex of everything on startup * `-v ~/music::r:ce2dsa:ce2tsr` does a full reindex of everything on startup
@@ -416,11 +338,11 @@ the same arguments can be set as volume flags, in addition to `d2d` and `d2t` fo
* `-v ~/music::r:cd2t` disables all `-e2t*` (tags), does not affect `-e2d*` * `-v ~/music::r:cd2t` disables all `-e2t*` (tags), does not affect `-e2d*`
note: note:
* `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those, unless there is a new copyparty version with new parsers and the release note says otherwise * `e2tsr` is probably always overkill, since `e2ds`/`e2dsa` would pick up any file modifications and `e2ts` would then reindex those
* the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher * the rescan button in the admin panel has no effect unless the volume has `-e2ds` or higher
you can choose to only index filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash` or the volume-flag `cdhash`, this has the following consequences: you can choose to only index filename/path/size/last-modified (and not the hash of the file contents) by setting `--no-hash` or the volume-flag `cdhash`, this has the following consequences:
* initial indexing is way faster, especially when the volume is on a network disk * initial indexing is way faster, especially when the volume is on a networked disk
* makes it impossible to [file-search](#file-search) * makes it impossible to [file-search](#file-search)
* if someone uploads the same file contents, the upload will not be detected as a dupe, so it will not get symlinked or rejected * if someone uploads the same file contents, the upload will not be detected as a dupe, so it will not get symlinked or rejected
@@ -453,17 +375,17 @@ tags that start with a `.` such as `.bpm` and `.dur`(ation) indicate numeric val
see the beautiful mess of a dictionary in [mtag.py](https://github.com/9001/copyparty/blob/master/copyparty/mtag.py) for the default mappings (should cover mp3,opus,flac,m4a,wav,aif,) see the beautiful mess of a dictionary in [mtag.py](https://github.com/9001/copyparty/blob/master/copyparty/mtag.py) for the default mappings (should cover mp3,opus,flac,m4a,wav,aif,)
`--no-mutagen` disables Mutagen and uses FFprobe instead, which... `--no-mutagen` disables mutagen and uses ffprobe instead, which...
* is about 20x slower than Mutagen * is about 20x slower than mutagen
* catches a few tags that Mutagen doesn't * catches a few tags that mutagen doesn't
* melodic key, video resolution, framerate, pixfmt * melodic key, video resolution, framerate, pixfmt
* avoids pulling any GPL code into copyparty * avoids pulling any GPL code into copyparty
* more importantly runs FFprobe on incoming files which is bad if your FFmpeg has a cve * more importantly runs ffprobe on incoming files which is bad if your ffmpeg has a cve
## file parser plugins ## file parser plugins
copyparty can invoke external programs to collect additional metadata for files using `mtp` (either as argument or volume flag), there is a default timeout of 30sec copyparty can invoke external programs to collect additional metadata for files using `mtp` (as argument or volume flag), there is a default timeout of 30sec
* `-mtp .bpm=~/bin/audio-bpm.py` will execute `~/bin/audio-bpm.py` with the audio file as argument 1 to provide the `.bpm` tag, if that does not exist in the audio metadata * `-mtp .bpm=~/bin/audio-bpm.py` will execute `~/bin/audio-bpm.py` with the audio file as argument 1 to provide the `.bpm` tag, if that does not exist in the audio metadata
* `-mtp key=f,t5,~/bin/audio-key.py` uses `~/bin/audio-key.py` to get the `key` tag, replacing any existing metadata tag (`f,`), aborting if it takes longer than 5sec (`t5,`) * `-mtp key=f,t5,~/bin/audio-key.py` uses `~/bin/audio-key.py` to get the `key` tag, replacing any existing metadata tag (`f,`), aborting if it takes longer than 5sec (`t5,`)
@@ -495,15 +417,13 @@ copyparty can invoke external programs to collect additional metadata for files
| send message | yep | yep | yep | yep | yep | yep | yep | yep | | send message | yep | yep | yep | yep | yep | yep | yep | yep |
| set sort order | - | yep | yep | yep | yep | yep | yep | yep | | set sort order | - | yep | yep | yep | yep | yep | yep | yep |
| zip selection | - | yep | yep | yep | yep | yep | yep | yep | | zip selection | - | yep | yep | yep | yep | yep | yep | yep |
| navpane | - | - | `*1` | yep | yep | yep | yep | yep | | directory tree | - | - | `*1` | yep | yep | yep | yep | yep |
| up2k | - | - | yep | yep | yep | yep | yep | yep | | up2k | - | - | yep | yep | yep | yep | yep | yep |
| icons work | - | - | yep | yep | yep | yep | yep | yep |
| markdown editor | - | - | yep | yep | yep | yep | yep | yep | | markdown editor | - | - | yep | yep | yep | yep | yep | yep |
| markdown viewer | - | - | yep | yep | yep | yep | yep | yep | | markdown viewer | - | - | yep | yep | yep | yep | yep | yep |
| play mp3/m4a | - | yep | yep | yep | yep | yep | yep | yep | | play mp3/m4a | - | yep | yep | yep | yep | yep | yep | yep |
| play ogg/opus | - | - | - | - | yep | yep | `*2` | yep | | play ogg/opus | - | - | - | - | yep | yep | `*2` | yep |
| thumbnail view | - | - | - | - | yep | yep | yep | yep |
| image viewer | - | - | - | - | yep | yep | yep | yep |
| **= feature =** | ie6 | ie9 | ie10 | ie11 | ff 52 | c 49 | iOS | Andr |
* internet explorer 6 to 8 behave the same * internet explorer 6 to 8 behave the same
* firefox 52 and chrome 49 are the last winxp versions * firefox 52 and chrome 49 are the last winxp versions
@@ -521,7 +441,7 @@ quick summary of more eccentric web-browsers trying to view a directory index:
| **w3m** (0.5.3/macports) | can browse, login, upload at 100kB/s, mkdir/msg | | **w3m** (0.5.3/macports) | can browse, login, upload at 100kB/s, mkdir/msg |
| **netsurf** (3.10/arch) | is basically ie6 with much better css (javascript has almost no effect) | | **netsurf** (3.10/arch) | is basically ie6 with much better css (javascript has almost no effect) |
| **ie4** and **netscape** 4.0 | can browse (text is yellow on white), upload with `?b=u` | | **ie4** and **netscape** 4.0 | can browse (text is yellow on white), upload with `?b=u` |
| **SerenityOS** (7e98457) | hits a page fault, works with `?b=u`, file upload not-impl | | **SerenityOS** (22d13d8) | hits a page fault, works with `?b=u`, file input not-impl, url params are multiplying |
# client examples # client examples
@@ -566,23 +486,6 @@ quick outline of the up2k protocol, see [uploading](#uploading) for the web-clie
* client does another handshake with the hashlist; server replies with OK or a list of chunks to reupload * client does another handshake with the hashlist; server replies with OK or a list of chunks to reupload
# performance
defaults are good for most cases, don't mind the `cannot efficiently use multiple CPU cores` message, it's very unlikely to be a problem
below are some tweaks roughly ordered by usefulness:
* `-q` disables logging and can help a bunch, even when combined with `-lo` to redirect logs to file
* `--http-only` or `--https-only` (unless you want to support both protocols) will reduce the delay before a new connection is established
* `--hist` pointing to a fast location (ssd) will make directory listings and searches faster when `-e2d` or `-e2t` is set
* `--no-hash` when indexing a network-disk if you don't care about the actual filehashes and only want the names/tags searchable
* `-j` enables multiprocessing (actual multithreading) and can make copyparty perform better in cpu-intensive workloads, for example:
* huge amount of short-lived connections
* really heavy traffic (downloads/uploads)
...however it adds an overhead to internal communication so it might be a net loss, see if it works 4 u
# dependencies # dependencies
* `jinja2` (is built into the SFX) * `jinja2` (is built into the SFX)
@@ -592,18 +495,18 @@ below are some tweaks roughly ordered by usefulness:
enable music tags: enable music tags:
* either `mutagen` (fast, pure-python, skips a few tags, makes copyparty GPL? idk) * either `mutagen` (fast, pure-python, skips a few tags, makes copyparty GPL? idk)
* or `ffprobe` (20x slower, more accurate, possibly dangerous depending on your distro and users) * or `FFprobe` (20x slower, more accurate, possibly dangerous depending on your distro and users)
enable thumbnails of images: enable image thumbnails:
* `Pillow` (requires py2.7 or py3.5+) * `Pillow` (requires py2.7 or py3.5+)
enable thumbnails of videos: enable video thumbnails:
* `ffmpeg` and `ffprobe` somewhere in `$PATH` * `ffmpeg` and `ffprobe` somewhere in `$PATH`
enable thumbnails of HEIF pictures: enable reading HEIF pictures:
* `pyheif-pillow-opener` (requires Linux or a C compiler) * `pyheif-pillow-opener` (requires Linux or a C compiler)
enable thumbnails of AVIF pictures: enable reading AVIF pictures:
* `pillow-avif-plugin` * `pillow-avif-plugin`
@@ -617,7 +520,7 @@ python -m pip install --user -U jinja2 mutagen Pillow
some bundled tools have copyleft dependencies, see [./bin/#mtag](bin/#mtag) some bundled tools have copyleft dependencies, see [./bin/#mtag](bin/#mtag)
these are standalone programs and will never be imported / evaluated by copyparty, and must be enabled through `-mtp` configs these are standalone programs and will never be imported / evaluated by copyparty
# sfx # sfx
@@ -633,10 +536,10 @@ pls note that `copyparty-sfx.sh` will fail if you rename `copyparty-sfx.py` to `
## sfx repack ## sfx repack
if you don't need all the features, you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except if you're on windows then you need msys2 or WSL) if you don't need all the features you can repack the sfx and save a bunch of space; all you need is an sfx and a copy of this repo (nothing else to download or build, except for either msys2 or WSL if you're on windows)
* `525k` size of original sfx.py as of v0.11.30 * `724K` original size as of v0.4.0
* `315k` after `./scripts/make-sfx.sh re no-ogv` * `256K` after `./scripts/make-sfx.sh re no-ogv`
* `223k` after `./scripts/make-sfx.sh re no-ogv no-cm` * `164K` after `./scripts/make-sfx.sh re no-ogv no-cm`
the features you can opt to drop are the features you can opt to drop are
* `ogv`.js, the opus/vorbis decoder which is needed by apple devices to play foss audio files * `ogv`.js, the opus/vorbis decoder which is needed by apple devices to play foss audio files
@@ -683,7 +586,7 @@ rm -rf copyparty/web/deps
curl -L https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py >x.py curl -L https://github.com/9001/copyparty/releases/latest/download/copyparty-sfx.py >x.py
python3 x.py -h python3 x.py -h
rm x.py rm x.py
mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/deps/ mv /tmp/pe-copyparty/copyparty/web/deps/ copyparty/web/
``` ```
then build the sfx using any of the following examples: then build the sfx using any of the following examples:
@@ -711,16 +614,13 @@ in the `scripts` folder:
roughly sorted by priority roughly sorted by priority
* hls framework for Someone Else to drop code into :^)
* readme.md as epilogue * readme.md as epilogue
## discarded ideas
* reduce up2k roundtrips * reduce up2k roundtrips
* start from a chunk index and just go * start from a chunk index and just go
* terminate client on bad data * terminate client on bad data
* not worth the effort, just throw enough conncetions at it
discarded ideas
* single sha512 across all up2k chunks? * single sha512 across all up2k chunks?
* crypto.subtle cannot into streaming, would have to use hashwasm, expensive * crypto.subtle cannot into streaming, would have to use hashwasm, expensive
* separate sqlite table per tag * separate sqlite table per tag
@@ -737,6 +637,3 @@ roughly sorted by priority
* nah * nah
* look into android thumbnail cache file format * look into android thumbnail cache file format
* absolutely not * absolutely not
* indexedDB for hashes, cfg enable/clear/sz, 2gb avail, ~9k for 1g, ~4k for 100m, 500k items before autoeviction
* blank hashlist when up-ok to skip handshake
* too many confusing side-effects

View File

@@ -345,7 +345,7 @@ class Gateway(object):
except: except:
pass pass
def sendreq(self, meth, path, headers, **kwargs): def sendreq(self, *args, headers={}, **kwargs):
if self.password: if self.password:
headers["Cookie"] = "=".join(["cppwd", self.password]) headers["Cookie"] = "=".join(["cppwd", self.password])
@@ -354,21 +354,21 @@ class Gateway(object):
if c.rx_path: if c.rx_path:
raise Exception() raise Exception()
c.request(meth, path, headers=headers, **kwargs) c.request(*list(args), headers=headers, **kwargs)
c.rx = c.getresponse() c.rx = c.getresponse()
return c return c
except: except:
tid = threading.current_thread().ident tid = threading.current_thread().ident
dbg( dbg(
"\033[1;37;44mbad conn {:x}\n {} {}\n {}\033[0m".format( "\033[1;37;44mbad conn {:x}\n {}\n {}\033[0m".format(
tid, meth, path, c.rx_path if c else "(null)" tid, " ".join(str(x) for x in args), c.rx_path if c else "(null)"
) )
) )
self.closeconn(c) self.closeconn(c)
c = self.getconn() c = self.getconn()
try: try:
c.request(meth, path, headers=headers, **kwargs) c.request(*list(args), headers=headers, **kwargs)
c.rx = c.getresponse() c.rx = c.getresponse()
return c return c
except: except:
@@ -386,7 +386,7 @@ class Gateway(object):
path = dewin(path) path = dewin(path)
web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots" web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots"
c = self.sendreq("GET", web_path, {}) c = self.sendreq("GET", web_path)
if c.rx.status != 200: if c.rx.status != 200:
self.closeconn(c) self.closeconn(c)
log( log(
@@ -440,7 +440,7 @@ class Gateway(object):
) )
) )
c = self.sendreq("GET", web_path, {"Range": hdr_range}) c = self.sendreq("GET", web_path, headers={"Range": hdr_range})
if c.rx.status != http.client.PARTIAL_CONTENT: if c.rx.status != http.client.PARTIAL_CONTENT:
self.closeconn(c) self.closeconn(c)
raise Exception( raise Exception(

View File

@@ -54,13 +54,10 @@ MACOS = platform.system() == "Darwin"
info = log = dbg = None info = log = dbg = None
print( print("{} v{} @ {}".format(
"{} v{} @ {}".format( platform.python_implementation(),
platform.python_implementation(), ".".join([str(x) for x in sys.version_info]),
".".join([str(x) for x in sys.version_info]), sys.executable))
sys.executable,
)
)
try: try:
@@ -302,14 +299,14 @@ class Gateway(object):
except: except:
pass pass
def sendreq(self, meth, path, headers, **kwargs): def sendreq(self, *args, headers={}, **kwargs):
tid = get_tid() tid = get_tid()
if self.password: if self.password:
headers["Cookie"] = "=".join(["cppwd", self.password]) headers["Cookie"] = "=".join(["cppwd", self.password])
try: try:
c = self.getconn(tid) c = self.getconn(tid)
c.request(meth, path, headers=headers, **kwargs) c.request(*list(args), headers=headers, **kwargs)
return c.getresponse() return c.getresponse()
except: except:
dbg("bad conn") dbg("bad conn")
@@ -317,7 +314,7 @@ class Gateway(object):
self.closeconn(tid) self.closeconn(tid)
try: try:
c = self.getconn(tid) c = self.getconn(tid)
c.request(meth, path, headers=headers, **kwargs) c.request(*list(args), headers=headers, **kwargs)
return c.getresponse() return c.getresponse()
except: except:
info("http connection failed:\n" + traceback.format_exc()) info("http connection failed:\n" + traceback.format_exc())
@@ -334,7 +331,7 @@ class Gateway(object):
path = dewin(path) path = dewin(path)
web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots&ls" web_path = self.quotep("/" + "/".join([self.web_root, path])) + "?dots&ls"
r = self.sendreq("GET", web_path, {}) r = self.sendreq("GET", web_path)
if r.status != 200: if r.status != 200:
self.closeconn() self.closeconn()
log( log(
@@ -371,7 +368,7 @@ class Gateway(object):
) )
) )
r = self.sendreq("GET", web_path, {"Range": hdr_range}) r = self.sendreq("GET", web_path, headers={"Range": hdr_range})
if r.status != http.client.PARTIAL_CONTENT: if r.status != http.client.PARTIAL_CONTENT:
self.closeconn() self.closeconn()
raise Exception( raise Exception(

View File

@@ -1,15 +1,7 @@
# when running copyparty behind a reverse proxy, # when running copyparty behind a reverse-proxy,
# the following arguments are recommended: # make sure that copyparty allows at least as many clients as the proxy does,
# # so run copyparty with -nc 512 if your nginx has the default limits
# -nc 512 important, see next paragraph # (worker_processes 1, worker_connections 512)
# --http-only lower latency on initial connection
# -i 127.0.0.1 only accept connections from nginx
#
# -nc must match or exceed the webserver's max number of concurrent clients;
# nginx default is 512 (worker_processes 1, worker_connections 512)
#
# you may also consider adding -j0 for CPU-intensive configurations
# (not that i can really think of any good examples)
upstream cpp { upstream cpp {
server 127.0.0.1:3923; server 127.0.0.1:3923;

View File

@@ -7,19 +7,11 @@
# you may want to: # you may want to:
# change '/usr/bin/python' to another interpreter # change '/usr/bin/python' to another interpreter
# change '/mnt::a' to another location or permission-set # change '/mnt::a' to another location or permission-set
#
# with `Type=notify`, copyparty will signal systemd when it is ready to
# accept connections; correctly delaying units depending on copyparty.
# But note that journalctl will get the timestamps wrong due to
# python disabling line-buffering, so messages are out-of-order:
# https://user-images.githubusercontent.com/241032/126040249-cb535cc7-c599-4931-a796-a5d9af691bad.png
[Unit] [Unit]
Description=copyparty file server Description=copyparty file server
[Service] [Service]
Type=notify
SyslogIdentifier=copyparty
ExecStart=/usr/bin/python3 /usr/local/bin/copyparty-sfx.py -q -v /mnt::a ExecStart=/usr/bin/python3 /usr/local/bin/copyparty-sfx.py -q -v /mnt::a
ExecStartPre=/bin/bash -c 'mkdir -p /run/tmpfiles.d/ && echo "x /tmp/pe-copyparty*" > /run/tmpfiles.d/copyparty.conf' ExecStartPre=/bin/bash -c 'mkdir -p /run/tmpfiles.d/ && echo "x /tmp/pe-copyparty*" > /run/tmpfiles.d/copyparty.conf'

View File

@@ -9,9 +9,6 @@ import os
PY2 = sys.version_info[0] == 2 PY2 = sys.version_info[0] == 2
if PY2: if PY2:
sys.dont_write_bytecode = True sys.dont_write_bytecode = True
unicode = unicode
else:
unicode = str
WINDOWS = False WINDOWS = False
if platform.system() == "Windows": if platform.system() == "Windows":

View File

@@ -20,10 +20,10 @@ import threading
import traceback import traceback
from textwrap import dedent from textwrap import dedent
from .__init__ import E, WINDOWS, VT100, PY2, unicode from .__init__ import E, WINDOWS, VT100, PY2
from .__version__ import S_VERSION, S_BUILD_DT, CODENAME from .__version__ import S_VERSION, S_BUILD_DT, CODENAME
from .svchub import SvcHub from .svchub import SvcHub
from .util import py_desc, align_tab, IMPLICATIONS from .util import py_desc, align_tab, IMPLICATIONS, alltrace
HAVE_SSL = True HAVE_SSL = True
try: try:
@@ -31,8 +31,6 @@ try:
except: except:
HAVE_SSL = False HAVE_SSL = False
printed = ""
class RiceFormatter(argparse.HelpFormatter): class RiceFormatter(argparse.HelpFormatter):
def _get_help_string(self, action): def _get_help_string(self, action):
@@ -63,15 +61,8 @@ class Dodge11874(RiceFormatter):
super(Dodge11874, self).__init__(*args, **kwargs) super(Dodge11874, self).__init__(*args, **kwargs)
def lprint(*a, **ka):
global printed
printed += " ".join(unicode(x) for x in a) + ka.get("end", "\n")
print(*a, **ka)
def warn(msg): def warn(msg):
lprint("\033[1mwarning:\033[0;33m {}\033[0m\n".format(msg)) print("\033[1mwarning:\033[0;33m {}\033[0m\n".format(msg))
def ensure_locale(): def ensure_locale():
@@ -82,7 +73,7 @@ def ensure_locale():
]: ]:
try: try:
locale.setlocale(locale.LC_ALL, x) locale.setlocale(locale.LC_ALL, x)
lprint("Locale:", x) print("Locale:", x)
break break
except: except:
continue continue
@@ -103,7 +94,7 @@ def ensure_cert():
try: try:
if filecmp.cmp(cert_cfg, cert_insec): if filecmp.cmp(cert_cfg, cert_insec):
lprint( print(
"\033[33m using default TLS certificate; https will be insecure." "\033[33m using default TLS certificate; https will be insecure."
+ "\033[36m\n certificate location: {}\033[0m\n".format(cert_cfg) + "\033[36m\n certificate location: {}\033[0m\n".format(cert_cfg)
) )
@@ -132,7 +123,7 @@ def configure_ssl_ver(al):
if "help" in sslver: if "help" in sslver:
avail = [terse_sslver(x[6:]) for x in flags] avail = [terse_sslver(x[6:]) for x in flags]
avail = " ".join(sorted(avail) + ["all"]) avail = " ".join(sorted(avail) + ["all"])
lprint("\navailable ssl/tls versions:\n " + avail) print("\navailable ssl/tls versions:\n " + avail)
sys.exit(0) sys.exit(0)
al.ssl_flags_en = 0 al.ssl_flags_en = 0
@@ -152,7 +143,7 @@ def configure_ssl_ver(al):
for k in ["ssl_flags_en", "ssl_flags_de"]: for k in ["ssl_flags_en", "ssl_flags_de"]:
num = getattr(al, k) num = getattr(al, k)
lprint("{}: {:8x} ({})".format(k, num, num)) print("{}: {:8x} ({})".format(k, num, num))
# think i need that beer now # think i need that beer now
@@ -169,13 +160,13 @@ def configure_ssl_ciphers(al):
try: try:
ctx.set_ciphers(al.ciphers) ctx.set_ciphers(al.ciphers)
except: except:
lprint("\n\033[1;31mfailed to set ciphers\033[0m\n") print("\n\033[1;31mfailed to set ciphers\033[0m\n")
if not hasattr(ctx, "get_ciphers"): if not hasattr(ctx, "get_ciphers"):
lprint("cannot read cipher list: openssl or python too old") print("cannot read cipher list: openssl or python too old")
else: else:
ciphers = [x["description"] for x in ctx.get_ciphers()] ciphers = [x["description"] for x in ctx.get_ciphers()]
lprint("\n ".join(["\nenabled ciphers:"] + align_tab(ciphers) + [""])) print("\n ".join(["\nenabled ciphers:"] + align_tab(ciphers) + [""]))
if is_help: if is_help:
sys.exit(0) sys.exit(0)
@@ -191,6 +182,16 @@ def sighandler(sig=None, frame=None):
print("\n".join(msg)) print("\n".join(msg))
def stackmon(fp, ival):
ctr = 0
while True:
ctr += 1
time.sleep(ival)
st = "{}, {}\n{}".format(ctr, time.time(), alltrace())
with open(fp, "wb") as f:
f.write(st.encode("utf-8", "replace"))
def run_argparse(argv, formatter): def run_argparse(argv, formatter):
ap = argparse.ArgumentParser( ap = argparse.ArgumentParser(
formatter_class=formatter, formatter_class=formatter,
@@ -199,30 +200,24 @@ def run_argparse(argv, formatter):
epilog=dedent( epilog=dedent(
""" """
-a takes username:password, -a takes username:password,
-v takes src:dst:perm1:perm2:permN:cflag1:cflag2:cflagN:... -v takes src:dst:permset:permset:cflag:cflag:...
where "perm" is "accesslevels,username1,username2,..." where "permset" is accesslevel followed by username (no separator)
and "cflag" is config flags to set on this volume and "cflag" is config flags to set on this volume
list of accesslevels:
"r" (read): list folder contents, download files
"w" (write): upload files; need "r" to see the uploads
"m" (move): move files and folders; need "w" at destination
"d" (delete): permanently delete files and folders
list of cflags: list of cflags:
"c,nodupe" rejects existing files (instead of symlinking them) "cnodupe" rejects existing files (instead of symlinking them)
"c,e2d" sets -e2d (all -e2* args can be set using ce2* cflags) "ce2d" sets -e2d (all -e2* args can be set using ce2* cflags)
"c,d2t" disables metadata collection, overrides -e2t* "cd2t" disables metadata collection, overrides -e2t*
"c,d2d" disables all database stuff, overrides -e2* "cd2d" disables all database stuff, overrides -e2*
example:\033[35m example:\033[35m
-a ed:hunter2 -v .::r:rw,ed -v ../inc:dump:w:rw,ed:c,nodupe \033[36m -a ed:hunter2 -v .::r:aed -v ../inc:dump:w:aed:cnodupe \033[36m
mount current directory at "/" with mount current directory at "/" with
* r (read-only) for everyone * r (read-only) for everyone
* rw (read+write) for ed * a (read+write) for ed
mount ../inc at "/dump" with mount ../inc at "/dump" with
* w (write-only) for everyone * w (write-only) for everyone
* rw (read+write) for ed * a (read+write) for ed
* reject duplicate files \033[0m * reject duplicate files \033[0m
if no accounts or volumes are configured, if no accounts or volumes are configured,
@@ -254,53 +249,46 @@ def run_argparse(argv, formatter):
), ),
) )
# fmt: off # fmt: off
u = unicode ap.add_argument("-c", metavar="PATH", type=str, action="append", help="add config file")
ap2 = ap.add_argument_group('general options') ap.add_argument("-nc", metavar="NUM", type=int, default=64, help="max num clients")
ap2.add_argument("-c", metavar="PATH", type=u, action="append", help="add config file") ap.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores")
ap2.add_argument("-nc", metavar="NUM", type=int, default=64, help="max num clients") ap.add_argument("-a", metavar="ACCT", type=str, action="append", help="add account, USER:PASS; example [ed:wark")
ap2.add_argument("-j", metavar="CORES", type=int, default=1, help="max num cpu cores") ap.add_argument("-v", metavar="VOL", type=str, action="append", help="add volume, SRC:DST:FLAG; example [.::r], [/mnt/nas/music:/music:r:aed")
ap2.add_argument("-a", metavar="ACCT", type=u, action="append", help="add account, USER:PASS; example [ed:wark") ap.add_argument("-ed", action="store_true", help="enable ?dots")
ap2.add_argument("-v", metavar="VOL", type=u, action="append", help="add volume, SRC:DST:FLAG; example [.::r], [/mnt/nas/music:/music:r:aed") ap.add_argument("-emp", action="store_true", help="enable markdown plugins")
ap2.add_argument("-ed", action="store_true", help="enable ?dots") ap.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate")
ap2.add_argument("-emp", action="store_true", help="enable markdown plugins") ap.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads")
ap2.add_argument("-mcr", metavar="SEC", type=int, default=60, help="md-editor mod-chk rate") ap.add_argument("--sparse", metavar="MiB", type=int, default=4, help="up2k min.size threshold (mswin-only)")
ap2.add_argument("--dotpart", action="store_true", help="dotfile incomplete uploads") ap.add_argument("--urlform", metavar="MODE", type=str, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
ap2.add_argument("--sparse", metavar="MiB", type=int, default=4, help="up2k min.size threshold (mswin-only)")
ap2.add_argument("--urlform", metavar="MODE", type=u, default="print,get", help="how to handle url-forms; examples: [stash], [save,get]")
ap2 = ap.add_argument_group('network options') ap2 = ap.add_argument_group('network options')
ap2.add_argument("-i", metavar="IP", type=u, default="0.0.0.0", help="ip to bind (comma-sep.)") ap2.add_argument("-i", metavar="IP", type=str, default="0.0.0.0", help="ip to bind (comma-sep.)")
ap2.add_argument("-p", metavar="PORT", type=u, default="3923", help="ports to bind (comma/range)") ap2.add_argument("-p", metavar="PORT", type=str, default="3923", help="ports to bind (comma/range)")
ap2.add_argument("--rproxy", metavar="DEPTH", type=int, default=1, help="which ip to keep; 0 = tcp, 1 = origin (first x-fwd), 2 = cloudflare, 3 = nginx, -1 = closest proxy") ap2.add_argument("--rproxy", metavar="DEPTH", type=int, default=1, help="which ip to keep; 0 = tcp, 1 = origin (first x-fwd), 2 = cloudflare, 3 = nginx, -1 = closest proxy")
ap2 = ap.add_argument_group('SSL/TLS options') ap2 = ap.add_argument_group('SSL/TLS options')
ap2.add_argument("--http-only", action="store_true", help="disable ssl/tls") ap2.add_argument("--http-only", action="store_true", help="disable ssl/tls")
ap2.add_argument("--https-only", action="store_true", help="disable plaintext") ap2.add_argument("--https-only", action="store_true", help="disable plaintext")
ap2.add_argument("--ssl-ver", metavar="LIST", type=u, help="set allowed ssl/tls versions; [help] shows available versions; default is what your python version considers safe") ap2.add_argument("--ssl-ver", metavar="LIST", type=str, help="set allowed ssl/tls versions; [help] shows available versions; default is what your python version considers safe")
ap2.add_argument("--ciphers", metavar="LIST", type=u, help="set allowed ssl/tls ciphers; [help] shows available ciphers") ap2.add_argument("--ciphers", metavar="LIST", help="set allowed ssl/tls ciphers; [help] shows available ciphers")
ap2.add_argument("--ssl-dbg", action="store_true", help="dump some tls info") ap2.add_argument("--ssl-dbg", action="store_true", help="dump some tls info")
ap2.add_argument("--ssl-log", metavar="PATH", type=u, help="log master secrets") ap2.add_argument("--ssl-log", metavar="PATH", help="log master secrets")
ap2 = ap.add_argument_group('opt-outs') ap2 = ap.add_argument_group('opt-outs')
ap2.add_argument("-nw", action="store_true", help="disable writes (benchmark)") ap2.add_argument("-nw", action="store_true", help="disable writes (benchmark)")
ap2.add_argument("--no-del", action="store_true", help="disable delete operations")
ap2.add_argument("--no-mv", action="store_true", help="disable move/rename operations")
ap2.add_argument("-nih", action="store_true", help="no info hostname") ap2.add_argument("-nih", action="store_true", help="no info hostname")
ap2.add_argument("-nid", action="store_true", help="no info disk-usage") ap2.add_argument("-nid", action="store_true", help="no info disk-usage")
ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar") ap2.add_argument("--no-zip", action="store_true", help="disable download as zip/tar")
ap2 = ap.add_argument_group('safety options') ap2 = ap.add_argument_group('safety options')
ap2.add_argument("--ls", metavar="U[,V[,F]]", type=u, help="scan all volumes; arguments USER,VOL,FLAGS; example [**,*,ln,p,r]") ap2.add_argument("--ls", metavar="U[,V[,F]]", help="scan all volumes; arguments USER,VOL,FLAGS; example [**,*,ln,p,r]")
ap2.add_argument("--salt", type=u, default="hunter2", help="up2k file-hash salt") ap2.add_argument("--salt", type=str, default="hunter2", help="up2k file-hash salt")
ap2 = ap.add_argument_group('logging options') ap2 = ap.add_argument_group('logging options')
ap2.add_argument("-q", action="store_true", help="quiet") ap2.add_argument("-q", action="store_true", help="quiet")
ap2.add_argument("-lo", metavar="PATH", type=u, help="logfile, example: cpp-%%Y-%%m%%d-%%H%%M%%S.txt.xz")
ap2.add_argument("--no-voldump", action="store_true", help="do not list volumes and permissions on startup")
ap2.add_argument("--log-conn", action="store_true", help="print tcp-server msgs") ap2.add_argument("--log-conn", action="store_true", help="print tcp-server msgs")
ap2.add_argument("--log-htp", action="store_true", help="print http-server threadpool scaling") ap2.add_argument("--ihead", metavar="HEADER", action='append', help="dump incoming header")
ap2.add_argument("--ihead", metavar="HEADER", type=u, action='append', help="dump incoming header") ap2.add_argument("--lf-url", metavar="RE", type=str, default=r"^/\.cpr/|\?th=[wj]$", help="dont log URLs matching")
ap2.add_argument("--lf-url", metavar="RE", type=u, default=r"^/\.cpr/|\?th=[wj]$", help="dont log URLs matching")
ap2 = ap.add_argument_group('admin panel options') ap2 = ap.add_argument_group('admin panel options')
ap2.add_argument("--no-rescan", action="store_true", help="disable ?scan (volume reindexing)") ap2.add_argument("--no-rescan", action="store_true", help="disable ?scan (volume reindexing)")
@@ -315,9 +303,9 @@ def run_argparse(argv, formatter):
ap2.add_argument("--th-no-webp", action="store_true", help="disable webp output") ap2.add_argument("--th-no-webp", action="store_true", help="disable webp output")
ap2.add_argument("--th-ff-jpg", action="store_true", help="force jpg for video thumbs") ap2.add_argument("--th-ff-jpg", action="store_true", help="force jpg for video thumbs")
ap2.add_argument("--th-poke", metavar="SEC", type=int, default=300, help="activity labeling cooldown") ap2.add_argument("--th-poke", metavar="SEC", type=int, default=300, help="activity labeling cooldown")
ap2.add_argument("--th-clean", metavar="SEC", type=int, default=43200, help="cleanup interval; 0=disabled") ap2.add_argument("--th-clean", metavar="SEC", type=int, default=43200, help="cleanup interval")
ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age") ap2.add_argument("--th-maxage", metavar="SEC", type=int, default=604800, help="max folder age")
ap2.add_argument("--th-covers", metavar="N,N", type=u, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat for") ap2.add_argument("--th-covers", metavar="N,N", type=str, default="folder.png,folder.jpg,cover.png,cover.jpg", help="folder thumbnails to stat for")
ap2 = ap.add_argument_group('database options') ap2 = ap.add_argument_group('database options')
ap2.add_argument("-e2d", action="store_true", help="enable up2k database") ap2.add_argument("-e2d", action="store_true", help="enable up2k database")
@@ -326,29 +314,27 @@ def run_argparse(argv, formatter):
ap2.add_argument("-e2t", action="store_true", help="enable metadata indexing") ap2.add_argument("-e2t", action="store_true", help="enable metadata indexing")
ap2.add_argument("-e2ts", action="store_true", help="enable metadata scanner, sets -e2t") ap2.add_argument("-e2ts", action="store_true", help="enable metadata scanner, sets -e2t")
ap2.add_argument("-e2tsr", action="store_true", help="rescan all metadata, sets -e2ts") ap2.add_argument("-e2tsr", action="store_true", help="rescan all metadata, sets -e2ts")
ap2.add_argument("--hist", metavar="PATH", type=u, help="where to store volume state") ap2.add_argument("--hist", metavar="PATH", type=str, help="where to store volume state")
ap2.add_argument("--no-hash", action="store_true", help="disable hashing during e2ds folder scans") ap2.add_argument("--no-hash", action="store_true", help="disable hashing during e2ds folder scans")
ap2.add_argument("--no-mutagen", action="store_true", help="use FFprobe for tags instead") ap2.add_argument("--no-mutagen", action="store_true", help="use ffprobe for tags instead")
ap2.add_argument("--no-mtag-mt", action="store_true", help="disable tag-read parallelism") ap2.add_argument("--no-mtag-mt", action="store_true", help="disable tag-read parallelism")
ap2.add_argument("--no-mtag-ff", action="store_true", help="never use FFprobe as tag reader") ap2.add_argument("-mtm", metavar="M=t,t,t", action="append", type=str, help="add/replace metadata mapping")
ap2.add_argument("--re-int", metavar="SEC", type=int, default=30, help="disk rescan check interval") ap2.add_argument("-mte", metavar="M,M,M", type=str, help="tags to index/display (comma-sep.)",
ap2.add_argument("--re-maxage", metavar="SEC", type=int, default=0, help="disk rescan volume interval (0=off)")
ap2.add_argument("-mtm", metavar="M=t,t,t", type=u, action="append", help="add/replace metadata mapping")
ap2.add_argument("-mte", metavar="M,M,M", type=u, help="tags to index/display (comma-sep.)",
default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,ac,vc,res,.fps") default="circle,album,.tn,artist,title,.bpm,key,.dur,.q,.vq,.aq,ac,vc,res,.fps")
ap2.add_argument("-mtp", metavar="M=[f,]bin", type=u, action="append", help="read tag M using bin") ap2.add_argument("-mtp", metavar="M=[f,]bin", action="append", type=str, help="read tag M using bin")
ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline") ap2.add_argument("--srch-time", metavar="SEC", type=int, default=30, help="search deadline")
ap2 = ap.add_argument_group('video streaming options')
ap2.add_argument("--vcr", action="store_true", help="enable video streaming")
ap2 = ap.add_argument_group('appearance options') ap2 = ap.add_argument_group('appearance options')
ap2.add_argument("--css-browser", metavar="L", type=u, help="URL to additional CSS to include") ap2.add_argument("--css-browser", metavar="L", help="URL to additional CSS to include")
ap2 = ap.add_argument_group('debug options') ap2 = ap.add_argument_group('debug options')
ap2.add_argument("--no-sendfile", action="store_true", help="disable sendfile") ap2.add_argument("--no-sendfile", action="store_true", help="disable sendfile")
ap2.add_argument("--no-scandir", action="store_true", help="disable scandir") ap2.add_argument("--no-scandir", action="store_true", help="disable scandir")
ap2.add_argument("--no-fastboot", action="store_true", help="wait for up2k indexing") ap2.add_argument("--no-fastboot", action="store_true", help="wait for up2k indexing")
ap2.add_argument("--no-htp", action="store_true", help="disable httpserver threadpool, create threads as-needed instead") ap2.add_argument("--stackmon", metavar="P,S", help="write stacktrace to Path every S second")
ap2.add_argument("--stackmon", metavar="P,S", type=u, help="write stacktrace to Path every S second")
ap2.add_argument("--log-thrs", metavar="SEC", type=float, help="list active threads every SEC")
return ap.parse_args(args=argv[1:]) return ap.parse_args(args=argv[1:])
# fmt: on # fmt: on
@@ -365,7 +351,7 @@ def main(argv=None):
desc = py_desc().replace("[", "\033[1;30m[") desc = py_desc().replace("[", "\033[1;30m[")
f = '\033[36mcopyparty v{} "\033[35m{}\033[36m" ({})\n{}\033[0m\n' f = '\033[36mcopyparty v{} "\033[35m{}\033[36m" ({})\n{}\033[0m\n'
lprint(f.format(S_VERSION, CODENAME, S_BUILD_DT, desc)) print(f.format(S_VERSION, CODENAME, S_BUILD_DT, desc))
ensure_locale() ensure_locale()
if HAVE_SSL: if HAVE_SSL:
@@ -379,7 +365,7 @@ def main(argv=None):
continue continue
msg = "\033[1;31mWARNING:\033[0;1m\n {} \033[0;33mwas replaced with\033[0;1m {} \033[0;33mand will be removed\n\033[0m" msg = "\033[1;31mWARNING:\033[0;1m\n {} \033[0;33mwas replaced with\033[0;1m {} \033[0;33mand will be removed\n\033[0m"
lprint(msg.format(dk, nk)) print(msg.format(dk, nk))
argv[idx] = nk argv[idx] = nk
time.sleep(2) time.sleep(2)
@@ -388,35 +374,15 @@ def main(argv=None):
except AssertionError: except AssertionError:
al = run_argparse(argv, Dodge11874) al = run_argparse(argv, Dodge11874)
nstrs = [] if al.stackmon:
anymod = False fp, f = al.stackmon.rsplit(",", 1)
for ostr in al.v or []: f = int(f)
mod = False t = threading.Thread(
oa = ostr.split(":") target=stackmon,
na = oa[:2] args=(fp, f),
for opt in oa[2:]: )
if re.match("c[^,]", opt): t.daemon = True
mod = True t.start()
na.append("c," + opt[2:])
elif re.sub("^[rwmd]*", "", opt) and "," not in opt:
mod = True
perm = opt[0]
if perm == "a":
perm = "rw"
na.append(perm + "," + opt[1:])
else:
na.append(opt)
nstr = ":".join(na)
nstrs.append(nstr if mod else ostr)
if mod:
msg = "\033[1;31mWARNING:\033[0;1m\n -v {} \033[0;33mwas replaced with\033[0;1m\n -v {} \n\033[0m"
lprint(msg.format(ostr, nstr))
anymod = True
if anymod:
al.v = nstrs
time.sleep(2)
# propagate implications # propagate implications
for k1, k2 in IMPLICATIONS: for k1, k2 in IMPLICATIONS:
@@ -453,7 +419,7 @@ def main(argv=None):
# signal.signal(signal.SIGINT, sighandler) # signal.signal(signal.SIGINT, sighandler)
SvcHub(al, argv, printed).run() SvcHub(al).run()
if __name__ == "__main__": if __name__ == "__main__":

View File

@@ -1,8 +1,8 @@
# coding: utf-8 # coding: utf-8
VERSION = (0, 12, 0) VERSION = (0, 11, 29)
CODENAME = "fil\033[33med" CODENAME = "the grid"
BUILD_DT = (2021, 7, 28) BUILD_DT = (2021, 6, 30)
S_VERSION = ".".join(map(str, VERSION)) S_VERSION = ".".join(map(str, VERSION))
S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT) S_BUILD_DT = "{0:04d}-{1:02d}-{2:02d}".format(*BUILD_DT)

View File

@@ -10,35 +10,19 @@ import hashlib
import threading import threading
from .__init__ import WINDOWS from .__init__ import WINDOWS
from .util import IMPLICATIONS, uncyg, undot, absreal, Pebkac, fsdec, fsenc, statdir from .util import IMPLICATIONS, uncyg, undot, Pebkac, fsdec, fsenc, statdir, nuprint
from .bos import bos
class AXS(object):
def __init__(self, uread=None, uwrite=None, umove=None, udel=None):
self.uread = {} if uread is None else {k: 1 for k in uread}
self.uwrite = {} if uwrite is None else {k: 1 for k in uwrite}
self.umove = {} if umove is None else {k: 1 for k in umove}
self.udel = {} if udel is None else {k: 1 for k in udel}
def __repr__(self):
return "AXS({})".format(
", ".join(
"{}={!r}".format(k, self.__dict__[k])
for k in "uread uwrite umove udel".split()
)
)
class VFS(object): class VFS(object):
"""single level in the virtual fs""" """single level in the virtual fs"""
def __init__(self, log, realpath, vpath, axs, flags): def __init__(self, realpath, vpath, uread=[], uwrite=[], uadm=[], flags={}):
self.log = log
self.realpath = realpath # absolute path on host filesystem self.realpath = realpath # absolute path on host filesystem
self.vpath = vpath # absolute path in the virtual filesystem self.vpath = vpath # absolute path in the virtual filesystem
self.axs = axs # type: AXS self.uread = uread # users who can read this
self.flags = flags # config options self.uwrite = uwrite # users who can write this
self.uadm = uadm # users who are regular admins
self.flags = flags # config switches
self.nodes = {} # child nodes self.nodes = {} # child nodes
self.histtab = None # all realpath->histpath self.histtab = None # all realpath->histpath
self.dbv = None # closest full/non-jump parent self.dbv = None # closest full/non-jump parent
@@ -46,23 +30,15 @@ class VFS(object):
if realpath: if realpath:
self.histpath = os.path.join(realpath, ".hist") # db / thumbcache self.histpath = os.path.join(realpath, ".hist") # db / thumbcache
self.all_vols = {vpath: self} # flattened recursive self.all_vols = {vpath: self} # flattened recursive
self.aread = {}
self.awrite = {}
self.amove = {}
self.adel = {}
else: else:
self.histpath = None self.histpath = None
self.all_vols = None self.all_vols = None
self.aread = None
self.awrite = None
self.amove = None
self.adel = None
def __repr__(self): def __repr__(self):
return "VFS({})".format( return "VFS({})".format(
", ".join( ", ".join(
"{}={!r}".format(k, self.__dict__[k]) "{}={!r}".format(k, self.__dict__[k])
for k in "realpath vpath axs flags".split() for k in "realpath vpath uread uwrite uadm flags".split()
) )
) )
@@ -86,10 +62,11 @@ class VFS(object):
return self.nodes[name].add(src, dst) return self.nodes[name].add(src, dst)
vn = VFS( vn = VFS(
self.log,
os.path.join(self.realpath, name) if self.realpath else None, os.path.join(self.realpath, name) if self.realpath else None,
"{}/{}".format(self.vpath, name).lstrip("/"), "{}/{}".format(self.vpath, name).lstrip("/"),
self.axs, self.uread,
self.uwrite,
self.uadm,
self._copy_flags(name), self._copy_flags(name),
) )
vn.dbv = self.dbv or self vn.dbv = self.dbv or self
@@ -102,7 +79,7 @@ class VFS(object):
# leaf does not exist; create and keep permissions blank # leaf does not exist; create and keep permissions blank
vp = "{}/{}".format(self.vpath, dst).lstrip("/") vp = "{}/{}".format(self.vpath, dst).lstrip("/")
vn = VFS(self.log, src, vp, AXS(), {}) vn = VFS(src, vp)
vn.dbv = self.dbv or self vn.dbv = self.dbv or self
self.nodes[dst] = vn self.nodes[dst] = vn
return vn return vn
@@ -142,32 +119,23 @@ class VFS(object):
return [self, vpath] return [self, vpath]
def can_access(self, vpath, uname): def can_access(self, vpath, uname):
# type: (str, str) -> tuple[bool, bool, bool, bool] """return [readable,writable]"""
"""can Read,Write,Move,Delete"""
vn, _ = self._find(vpath) vn, _ = self._find(vpath)
c = vn.axs
return [ return [
uname in c.uread or "*" in c.uread, uname in vn.uread or "*" in vn.uread,
uname in c.uwrite or "*" in c.uwrite, uname in vn.uwrite or "*" in vn.uwrite,
uname in c.umove or "*" in c.umove,
uname in c.udel or "*" in c.udel,
] ]
def get(self, vpath, uname, will_read, will_write, will_move=False, will_del=False): def get(self, vpath, uname, will_read, will_write):
# type: (str, str, bool, bool, bool, bool) -> tuple[VFS, str] # type: (str, str, bool, bool) -> tuple[VFS, str]
"""returns [vfsnode,fs_remainder] if user has the requested permissions""" """returns [vfsnode,fs_remainder] if user has the requested permissions"""
vn, rem = self._find(vpath) vn, rem = self._find(vpath)
c = vn.axs
for req, d, msg in [ if will_read and (uname not in vn.uread and "*" not in vn.uread):
[will_read, c.uread, "read"], raise Pebkac(403, "you don't have read-access for this location")
[will_write, c.uwrite, "write"],
[will_move, c.umove, "move"], if will_write and (uname not in vn.uwrite and "*" not in vn.uwrite):
[will_del, c.udel, "delete"], raise Pebkac(403, "you don't have write-access for this location")
]:
if req and (uname not in d and "*" not in d):
m = "you don't have {}-access for this location"
raise Pebkac(403, m.format(msg))
return vn, rem return vn, rem
@@ -180,58 +148,68 @@ class VFS(object):
vrem = "/".join([x for x in vrem if x]) vrem = "/".join([x for x in vrem if x])
return dbv, vrem return dbv, vrem
def canonical(self, rem, resolve=True): def canonical(self, rem):
"""returns the canonical path (fully-resolved absolute fs path)""" """returns the canonical path (fully-resolved absolute fs path)"""
rp = self.realpath rp = self.realpath
if rem: if rem:
rp += "/" + rem rp += "/" + rem
return absreal(rp) if resolve else rp try:
return fsdec(os.path.realpath(fsenc(rp)))
except:
if not WINDOWS:
raise
def ls(self, rem, uname, scandir, permsets, lstat=False): # cpython bug introduced in 3.8, still exists in 3.9.1;
# type: (str, str, bool, list[list[bool]], bool) -> tuple[str, str, dict[str, VFS]] # some win7sp1 and win10:20H2 boxes cannot realpath a
# networked drive letter such as b"n:" or b"n:\\"
#
# requirements to trigger:
# * bytestring (not unicode str)
# * just the drive letter (subfolders are ok)
# * networked drive (regular disks and vmhgfs are ok)
# * on an enterprise network (idk, cannot repro with samba)
#
# hits the following exceptions in succession:
# * access denied at L601: "path = _getfinalpathname(path)"
# * "cant concat str to bytes" at L621: "return path + tail"
#
return os.path.realpath(rp)
def ls(self, rem, uname, scandir, incl_wo=False, lstat=False):
# type: (str, str, bool, bool, bool) -> tuple[str, str, dict[str, VFS]]
"""return user-readable [fsdir,real,virt] items at vpath""" """return user-readable [fsdir,real,virt] items at vpath"""
virt_vis = {} # nodes readable by user virt_vis = {} # nodes readable by user
abspath = self.canonical(rem) abspath = self.canonical(rem)
real = list(statdir(self.log, scandir, lstat, abspath)) real = list(statdir(nuprint, scandir, lstat, abspath))
real.sort() real.sort()
if not rem: if not rem:
# no vfs nodes in the list of real inodes
real = [x for x in real if x[0] not in self.nodes]
for name, vn2 in sorted(self.nodes.items()): for name, vn2 in sorted(self.nodes.items()):
ok = False ok = uname in vn2.uread or "*" in vn2.uread
axs = vn2.axs
axs = [axs.uread, axs.uwrite, axs.umove, axs.udel] if not ok and incl_wo:
for pset in permsets: ok = uname in vn2.uwrite or "*" in vn2.uwrite
ok = True
for req, lst in zip(pset, axs):
if req and uname not in lst and "*" not in lst:
ok = False
if ok:
break
if ok: if ok:
virt_vis[name] = vn2 virt_vis[name] = vn2
# no vfs nodes in the list of real inodes
real = [x for x in real if x[0] not in self.nodes]
return [abspath, real, virt_vis] return [abspath, real, virt_vis]
def walk(self, rel, rem, seen, uname, permsets, dots, scandir, lstat): def walk(self, rel, rem, seen, uname, dots, scandir, lstat):
""" """
recursively yields from ./rem; recursively yields from ./rem;
rel is a unix-style user-defined vpath (not vfs-related) rel is a unix-style user-defined vpath (not vfs-related)
""" """
fsroot, vfs_ls, vfs_virt = self.ls(rem, uname, scandir, permsets, lstat=lstat) fsroot, vfs_ls, vfs_virt = self.ls(
dbv, vrem = self.get_dbv(rem) rem, uname, scandir, incl_wo=False, lstat=lstat
)
if ( if seen and not fsroot.startswith(seen[-1]) and fsroot in seen:
seen print("bailing from symlink loop,\n {}\n {}".format(seen[-1], fsroot))
and (not fsroot.startswith(seen[-1]) or fsroot == seen[-1])
and fsroot in seen
):
m = "bailing from symlink loop,\n prev: {}\n curr: {}\n from: {}/{}"
self.log("vfs.walk", m.format(seen[-1], fsroot, self.vpath, rem), 3)
return return
seen = seen[:] + [fsroot] seen = seen[:] + [fsroot]
@@ -241,7 +219,7 @@ class VFS(object):
rfiles.sort() rfiles.sort()
rdirs.sort() rdirs.sort()
yield dbv, vrem, rel, fsroot, rfiles, rdirs, vfs_virt yield rel, fsroot, rfiles, rdirs, vfs_virt
for rdir, _ in rdirs: for rdir, _ in rdirs:
if not dots and rdir.startswith("."): if not dots and rdir.startswith("."):
@@ -249,7 +227,7 @@ class VFS(object):
wrel = (rel + "/" + rdir).lstrip("/") wrel = (rel + "/" + rdir).lstrip("/")
wrem = (rem + "/" + rdir).lstrip("/") wrem = (rem + "/" + rdir).lstrip("/")
for x in self.walk(wrel, wrem, seen, uname, permsets, dots, scandir, lstat): for x in self.walk(wrel, wrem, seen, uname, dots, scandir, lstat):
yield x yield x
for n, vfs in sorted(vfs_virt.items()): for n, vfs in sorted(vfs_virt.items()):
@@ -257,19 +235,16 @@ class VFS(object):
continue continue
wrel = (rel + "/" + n).lstrip("/") wrel = (rel + "/" + n).lstrip("/")
for x in vfs.walk(wrel, "", seen, uname, permsets, dots, scandir, lstat): for x in vfs.walk(wrel, "", seen, uname, dots, scandir, lstat):
yield x yield x
def zipgen(self, vrem, flt, uname, dots, scandir): def zipgen(self, vrem, flt, uname, dots, scandir):
if flt: if flt:
flt = {k: True for k in flt} flt = {k: True for k in flt}
f1 = "{0}.hist{0}up2k.".format(os.sep) for vpath, apath, files, rd, vd in self.walk(
f2a = os.sep + "dir.txt" "", vrem, [], uname, dots, scandir, False
f2b = "{0}.hist{0}".format(os.sep) ):
g = self.walk("", vrem, [], uname, [[True]], dots, scandir, False)
for _, _, vpath, apath, files, rd, vd in g:
if flt: if flt:
files = [x for x in files if x[0] in flt] files = [x for x in files if x[0] in flt]
@@ -300,15 +275,25 @@ class VFS(object):
del vd[x] del vd[x]
# up2k filetring based on actual abspath # up2k filetring based on actual abspath
files = [ files = [x for x in files if "{0}.hist{0}up2k.".format(os.sep) not in x[1]]
x
for x in files
if f1 not in x[1] and (not x[1].endswith(f2a) or f2b not in x[1])
]
for f in [{"vp": v, "ap": a, "st": n[1]} for v, a, n in files]: for f in [{"vp": v, "ap": a, "st": n[1]} for v, a, n in files]:
yield f yield f
def user_tree(self, uname, readable, writable, admin):
is_readable = False
if uname in self.uread or "*" in self.uread:
readable.append(self.vpath)
is_readable = True
if uname in self.uwrite or "*" in self.uwrite:
writable.append(self.vpath)
if is_readable:
admin.append(self.vpath)
for _, vn in sorted(self.nodes.items()):
vn.user_tree(uname, readable, writable, admin)
class AuthSrv(object): class AuthSrv(object):
"""verifies users against given paths""" """verifies users against given paths"""
@@ -341,8 +326,7 @@ class AuthSrv(object):
yield prev, True yield prev, True
def _parse_config_file(self, fd, acct, daxs, mflags, mount): def _parse_config_file(self, fd, user, mread, mwrite, madm, mflags, mount):
# type: (any, str, dict[str, AXS], any, str) -> None
vol_src = None vol_src = None
vol_dst = None vol_dst = None
self.line_ctr = 0 self.line_ctr = 0
@@ -358,7 +342,7 @@ class AuthSrv(object):
if vol_src is None: if vol_src is None:
if ln.startswith("u "): if ln.startswith("u "):
u, p = ln[2:].split(":", 1) u, p = ln[2:].split(":", 1)
acct[u] = p user[u] = p
else: else:
vol_src = ln vol_src = ln
continue continue
@@ -369,49 +353,50 @@ class AuthSrv(object):
raise Exception('invalid mountpoint "{}"'.format(vol_dst)) raise Exception('invalid mountpoint "{}"'.format(vol_dst))
# cfg files override arguments and previous files # cfg files override arguments and previous files
vol_src = bos.path.abspath(vol_src) vol_src = fsdec(os.path.abspath(fsenc(vol_src)))
vol_dst = vol_dst.strip("/") vol_dst = vol_dst.strip("/")
mount[vol_dst] = vol_src mount[vol_dst] = vol_src
daxs[vol_dst] = AXS() mread[vol_dst] = []
mwrite[vol_dst] = []
madm[vol_dst] = []
mflags[vol_dst] = {} mflags[vol_dst] = {}
continue continue
try: if len(ln) > 1:
lvl, uname = ln.split(" ", 1) lvl, uname = ln.split(" ")
except: else:
lvl = ln lvl = ln
uname = "*" uname = "*"
if lvl == "a": self._read_vol_str(
m = "WARNING (config-file): permission flag 'a' is deprecated; please use 'rw' instead" lvl,
self.log(m, 1) uname,
mread[vol_dst],
mwrite[vol_dst],
madm[vol_dst],
mflags[vol_dst],
)
self._read_vol_str(lvl, uname, daxs[vol_dst], mflags[vol_dst]) def _read_vol_str(self, lvl, uname, mr, mw, ma, mf):
def _read_vol_str(self, lvl, uname, axs, flags):
# type: (str, str, AXS, any) -> None
if lvl == "c": if lvl == "c":
cval = True cval = True
if "=" in uname: if "=" in uname:
uname, cval = uname.split("=", 1) uname, cval = uname.split("=", 1)
self._read_volflag(flags, uname, cval, False) self._read_volflag(mf, uname, cval, False)
return return
if uname == "": if uname == "":
uname = "*" uname = "*"
if "r" in lvl: if lvl in "ra":
axs.uread[uname] = 1 mr.append(uname)
if "w" in lvl: if lvl in "wa":
axs.uwrite[uname] = 1 mw.append(uname)
if "m" in lvl: if lvl == "a":
axs.umove[uname] = 1 ma.append(uname)
if "d" in lvl:
axs.udel[uname] = 1
def _read_volflag(self, flags, name, value, is_list): def _read_volflag(self, flags, name, value, is_list):
if name not in ["mtp"]: if name not in ["mtp"]:
@@ -433,24 +418,21 @@ class AuthSrv(object):
before finally building the VFS before finally building the VFS
""" """
acct = {} # username:password user = {} # username:password
daxs = {} # type: dict[str, AXS] mread = {} # mountpoint:[username]
mwrite = {} # mountpoint:[username]
madm = {} # mountpoint:[username]
mflags = {} # mountpoint:[flag] mflags = {} # mountpoint:[flag]
mount = {} # dst:src (mountpoint:realpath) mount = {} # dst:src (mountpoint:realpath)
if self.args.a: if self.args.a:
# list of username:password # list of username:password
for x in self.args.a: for u, p in [x.split(":", 1) for x in self.args.a]:
try: user[u] = p
u, p = x.split(":", 1)
acct[u] = p
except:
m = '\n invalid value "{}" for argument -a, must be username:password'
raise Exception(m.format(x))
if self.args.v: if self.args.v:
# list of src:dst:permset:permset:... # list of src:dst:permset:permset:...
# permset is <rwmd>[,username][,username] or <c>,<flag>[=args] # permset is [rwa]username or [c]flag
for v_str in self.args.v: for v_str in self.args.v:
m = self.re_vol.match(v_str) m = self.re_vol.match(v_str)
if not m: if not m:
@@ -461,41 +443,49 @@ class AuthSrv(object):
src = uncyg(src) src = uncyg(src)
# print("\n".join([src, dst, perms])) # print("\n".join([src, dst, perms]))
src = bos.path.abspath(src) src = fsdec(os.path.abspath(fsenc(src)))
dst = dst.strip("/") dst = dst.strip("/")
mount[dst] = src mount[dst] = src
daxs[dst] = AXS() mread[dst] = []
mwrite[dst] = []
madm[dst] = []
mflags[dst] = {} mflags[dst] = {}
for x in perms.split(":"): perms = perms.split(":")
lvl, uname = x.split(",", 1) if "," in x else [x, ""] for (lvl, uname) in [[x[0], x[1:]] for x in perms]:
self._read_vol_str(lvl, uname, daxs[dst], mflags[dst]) self._read_vol_str(
lvl, uname, mread[dst], mwrite[dst], madm[dst], mflags[dst]
)
if self.args.c: if self.args.c:
for cfg_fn in self.args.c: for cfg_fn in self.args.c:
with open(cfg_fn, "rb") as f: with open(cfg_fn, "rb") as f:
try: try:
self._parse_config_file(f, acct, daxs, mflags, mount) self._parse_config_file(
f, user, mread, mwrite, madm, mflags, mount
)
except: except:
m = "\n\033[1;31m\nerror in config file {} on line {}:\n\033[0m" m = "\n\033[1;31m\nerror in config file {} on line {}:\n\033[0m"
self.log(m.format(cfg_fn, self.line_ctr), 1) print(m.format(cfg_fn, self.line_ctr))
raise raise
# case-insensitive; normalize # case-insensitive; normalize
if WINDOWS: if WINDOWS:
cased = {} cased = {}
for k, v in mount.items(): for k, v in mount.items():
cased[k] = absreal(v) try:
cased[k] = fsdec(os.path.realpath(fsenc(v)))
except:
cased[k] = v
mount = cased mount = cased
if not mount: if not mount:
# -h says our defaults are CWD at root and read/write for everyone # -h says our defaults are CWD at root and read/write for everyone
axs = AXS(["*"], ["*"], None, None) vfs = VFS(os.path.abspath("."), "", ["*"], ["*"])
vfs = VFS(self.log_func, bos.path.abspath("."), "", axs, {})
elif "" not in mount: elif "" not in mount:
# there's volumes but no root; make root inaccessible # there's volumes but no root; make root inaccessible
vfs = VFS(self.log_func, None, "", AXS(), {}) vfs = VFS(None, "")
vfs.flags["d2d"] = True vfs.flags["d2d"] = True
maxdepth = 0 maxdepth = 0
@@ -506,34 +496,26 @@ class AuthSrv(object):
if dst == "": if dst == "":
# rootfs was mapped; fully replaces the default CWD vfs # rootfs was mapped; fully replaces the default CWD vfs
vfs = VFS(self.log_func, mount[dst], dst, daxs[dst], mflags[dst]) vfs = VFS(
mount[dst], dst, mread[dst], mwrite[dst], madm[dst], mflags[dst]
)
continue continue
v = vfs.add(mount[dst], dst) v = vfs.add(mount[dst], dst)
v.axs = daxs[dst] v.uread = mread[dst]
v.uwrite = mwrite[dst]
v.uadm = madm[dst]
v.flags = mflags[dst] v.flags = mflags[dst]
v.dbv = None v.dbv = None
vfs.all_vols = {} vfs.all_vols = {}
vfs.get_all_vols(vfs.all_vols) vfs.get_all_vols(vfs.all_vols)
for perm in "read write move del".split():
axs_key = "u" + perm
unames = ["*"] + list(acct.keys())
umap = {x: [] for x in unames}
for usr in unames:
for mp, vol in vfs.all_vols.items():
if usr in getattr(vol.axs, axs_key):
umap[usr].append(mp)
setattr(vfs, "a" + perm, umap)
all_users = {}
missing_users = {} missing_users = {}
for axs in daxs.values(): for d in [mread, mwrite]:
for d in [axs.uread, axs.uwrite, axs.umove, axs.udel]: for _, ul in d.items():
for usr in d.keys(): for usr in ul:
all_users[usr] = 1 if usr != "*" and usr not in user:
if usr != "*" and usr not in acct:
missing_users[usr] = 1 missing_users[usr] = 1
if missing_users: if missing_users:
@@ -557,7 +539,10 @@ class AuthSrv(object):
elif self.args.hist: elif self.args.hist:
for nch in range(len(hid)): for nch in range(len(hid)):
hpath = os.path.join(self.args.hist, hid[: nch + 1]) hpath = os.path.join(self.args.hist, hid[: nch + 1])
bos.makedirs(hpath) try:
os.makedirs(hpath)
except:
pass
powner = os.path.join(hpath, "owner.txt") powner = os.path.join(hpath, "owner.txt")
try: try:
@@ -577,9 +562,9 @@ class AuthSrv(object):
vol.histpath = hpath vol.histpath = hpath
break break
vol.histpath = absreal(vol.histpath) vol.histpath = os.path.realpath(vol.histpath)
if vol.dbv: if vol.dbv:
if bos.path.exists(os.path.join(vol.histpath, "up2k.db")): if os.path.exists(os.path.join(vol.histpath, "up2k.db")):
promote.append(vol) promote.append(vol)
vol.dbv = None vol.dbv = None
else: else:
@@ -605,7 +590,7 @@ class AuthSrv(object):
all_mte = {} all_mte = {}
errors = False errors = False
for vol in vfs.all_vols.values(): for vol in vfs.all_vols.values():
if (self.args.e2ds and vol.axs.uwrite) or self.args.e2dsa: if (self.args.e2ds and vol.uwrite) or self.args.e2dsa:
vol.flags["e2ds"] = True vol.flags["e2ds"] = True
if self.args.e2d or "e2ds" in vol.flags: if self.args.e2d or "e2ds" in vol.flags:
@@ -694,27 +679,6 @@ class AuthSrv(object):
vfs.bubble_flags() vfs.bubble_flags()
m = "volumes and permissions:\n"
for v in vfs.all_vols.values():
if not self.warn_anonwrite:
break
m += '\n\033[36m"/{}" \033[33m{}\033[0m'.format(v.vpath, v.realpath)
for txt, attr in [
[" read", "uread"],
[" write", "uwrite"],
[" move", "umove"],
["delete", "udel"],
]:
u = list(sorted(getattr(v.axs, attr).keys()))
u = ", ".join("\033[35meverybody\033[0m" if x == "*" else x for x in u)
u = u if u else "\033[36m--none--\033[0m"
m += "\n| {}: {}".format(txt, u)
m += "\n"
if self.warn_anonwrite and not self.args.no_voldump:
self.log(m)
try: try:
v, _ = vfs.get("/", "*", False, True) v, _ = vfs.get("/", "*", False, True)
if self.warn_anonwrite and os.getcwd() == v.realpath: if self.warn_anonwrite and os.getcwd() == v.realpath:
@@ -726,14 +690,17 @@ class AuthSrv(object):
with self.mutex: with self.mutex:
self.vfs = vfs self.vfs = vfs
self.acct = acct self.user = user
self.iacct = {v: k for k, v in acct.items()} self.iuser = {v: k for k, v in user.items()}
self.re_pwd = None self.re_pwd = None
pwds = [re.escape(x) for x in self.iacct.keys()] pwds = [re.escape(x) for x in self.iuser.keys()]
if pwds: if pwds:
self.re_pwd = re.compile("=(" + "|".join(pwds) + ")([]&; ]|$)") self.re_pwd = re.compile("=(" + "|".join(pwds) + ")([]&; ]|$)")
# import pprint
# pprint.pprint({"usr": user, "rd": mread, "wr": mwrite, "mnt": mount})
def dbg_ls(self): def dbg_ls(self):
users = self.args.ls users = self.args.ls
vols = "*" vols = "*"
@@ -751,12 +718,12 @@ class AuthSrv(object):
pass pass
if users == "**": if users == "**":
users = list(self.acct.keys()) + ["*"] users = list(self.user.keys()) + ["*"]
else: else:
users = [users] users = [users]
for u in users: for u in users:
if u not in self.acct and u != "*": if u not in self.user and u != "*":
raise Exception("user not found: " + u) raise Exception("user not found: " + u)
if vols == "*": if vols == "*":
@@ -772,10 +739,8 @@ class AuthSrv(object):
raise Exception("volume not found: " + v) raise Exception("volume not found: " + v)
self.log({"users": users, "vols": vols, "flags": flags}) self.log({"users": users, "vols": vols, "flags": flags})
m = "/{}: read({}) write({}) move({}) del({})"
for k, v in self.vfs.all_vols.items(): for k, v in self.vfs.all_vols.items():
vc = v.axs self.log("/{}: read({}) write({})".format(k, v.uread, v.uwrite))
self.log(m.format(k, vc.uread, vc.uwrite, vc.umove, vc.udel))
flag_v = "v" in flags flag_v = "v" in flags
flag_ln = "ln" in flags flag_ln = "ln" in flags
@@ -789,15 +754,13 @@ class AuthSrv(object):
for u in users: for u in users:
self.log("checking /{} as {}".format(v, u)) self.log("checking /{} as {}".format(v, u))
try: try:
vn, _ = self.vfs.get(v, u, True, False, False, False) vn, _ = self.vfs.get(v, u, True, False)
except: except:
continue continue
atop = vn.realpath atop = vn.realpath
g = vn.walk( g = vn.walk("", "", [], u, True, not self.args.no_scandir, False)
"", "", [], u, True, [[True]], not self.args.no_scandir, False for vpath, apath, files, _, _ in g:
)
for _, _, vpath, apath, files, _, _ in g:
fnames = [n[0] for n in files] fnames = [n[0] for n in files]
vpaths = [vpath + "/" + n for n in fnames] if vpath else fnames vpaths = [vpath + "/" + n for n in fnames] if vpath else fnames
vpaths = [vtop + x for x in vpaths] vpaths = [vtop + x for x in vpaths]
@@ -817,7 +780,7 @@ class AuthSrv(object):
msg = [x[1] for x in files] msg = [x[1] for x in files]
if msg: if msg:
self.log("\n" + "\n".join(msg)) nuprint("\n".join(msg))
if n_bads and flag_p: if n_bads and flag_p:
raise Exception("found symlink leaving volume, and strict is set") raise Exception("found symlink leaving volume, and strict is set")

View File

@@ -1,59 +0,0 @@
# coding: utf-8
from __future__ import print_function, unicode_literals
import os
from ..util import fsenc, fsdec
from . import path
# grep -hRiE '(^|[^a-zA-Z_\.-])os\.' . | gsed -r 's/ /\n/g;s/\(/(\n/g' | grep -hRiE '(^|[^a-zA-Z_\.-])os\.' | sort | uniq -c
# printf 'os\.(%s)' "$(grep ^def bos/__init__.py | gsed -r 's/^def //;s/\(.*//' | tr '\n' '|' | gsed -r 's/.$//')"
def chmod(p, mode):
return os.chmod(fsenc(p), mode)
def listdir(p="."):
return [fsdec(x) for x in os.listdir(fsenc(p))]
def lstat(p):
return os.lstat(fsenc(p))
def makedirs(name, mode=0o755, exist_ok=True):
bname = fsenc(name)
try:
os.makedirs(bname, mode=mode)
except:
if not exist_ok or not os.path.isdir(bname):
raise
def mkdir(p, mode=0o755):
return os.mkdir(fsenc(p), mode=mode)
def rename(src, dst):
return os.rename(fsenc(src), fsenc(dst))
def replace(src, dst):
return os.replace(fsenc(src), fsenc(dst))
def rmdir(p):
return os.rmdir(fsenc(p))
def stat(p):
return os.stat(fsenc(p))
def unlink(p):
return os.unlink(fsenc(p))
def utime(p, times=None):
return os.utime(fsenc(p), times)

View File

@@ -1,33 +0,0 @@
# coding: utf-8
from __future__ import print_function, unicode_literals
import os
from ..util import fsenc, fsdec
def abspath(p):
return fsdec(os.path.abspath(fsenc(p)))
def exists(p):
return os.path.exists(fsenc(p))
def getmtime(p):
return os.path.getmtime(fsenc(p))
def getsize(p):
return os.path.getsize(fsenc(p))
def isdir(p):
return os.path.isdir(fsenc(p))
def islink(p):
return os.path.islink(fsenc(p))
def realpath(p):
return fsdec(os.path.realpath(fsenc(p)))

View File

@@ -4,11 +4,17 @@ from __future__ import print_function, unicode_literals
import time import time
import threading import threading
from .__init__ import PY2, WINDOWS, VT100
from .broker_util import try_exec from .broker_util import try_exec
from .broker_mpw import MpWorker from .broker_mpw import MpWorker
from .util import mp from .util import mp
if PY2 and not WINDOWS:
from multiprocessing.reduction import ForkingPickler
from StringIO import StringIO as MemesIO # pylint: disable=import-error
class BrokerMp(object): class BrokerMp(object):
"""external api; manages MpWorkers""" """external api; manages MpWorkers"""
@@ -22,19 +28,24 @@ class BrokerMp(object):
self.retpend_mutex = threading.Lock() self.retpend_mutex = threading.Lock()
self.mutex = threading.Lock() self.mutex = threading.Lock()
self.num_workers = self.args.j or mp.cpu_count() cores = self.args.j
self.log("broker", "booting {} subprocesses".format(self.num_workers)) if not cores:
for n in range(1, self.num_workers + 1): cores = mp.cpu_count()
self.log("broker", "booting {} subprocesses".format(cores))
for n in range(cores):
q_pend = mp.Queue(1) q_pend = mp.Queue(1)
q_yield = mp.Queue(64) q_yield = mp.Queue(64)
proc = mp.Process(target=MpWorker, args=(q_pend, q_yield, self.args, n)) proc = mp.Process(target=MpWorker, args=(q_pend, q_yield, self.args, n))
proc.q_pend = q_pend proc.q_pend = q_pend
proc.q_yield = q_yield proc.q_yield = q_yield
proc.nid = n
proc.clients = {} proc.clients = {}
proc.workload = 0
thr = threading.Thread( thr = threading.Thread(
target=self.collector, args=(proc,), name="mp-sink-{}".format(n) target=self.collector, args=(proc,), name="mp-collector"
) )
thr.daemon = True thr.daemon = True
thr.start() thr.start()
@@ -42,6 +53,13 @@ class BrokerMp(object):
self.procs.append(proc) self.procs.append(proc)
proc.start() proc.start()
if not self.args.q:
thr = threading.Thread(
target=self.debug_load_balancer, name="mp-dbg-loadbalancer"
)
thr.daemon = True
thr.start()
def shutdown(self): def shutdown(self):
self.log("broker", "shutting down") self.log("broker", "shutting down")
for n, proc in enumerate(self.procs): for n, proc in enumerate(self.procs):
@@ -71,6 +89,20 @@ class BrokerMp(object):
if dest == "log": if dest == "log":
self.log(*args) self.log(*args)
elif dest == "workload":
with self.mutex:
proc.workload = args[0]
elif dest == "httpdrop":
addr = args[0]
with self.mutex:
del proc.clients[addr]
if not proc.clients:
proc.workload = 0
self.hub.tcpsrv.num_clients.add(-1)
elif dest == "retq": elif dest == "retq":
# response from previous ipc call # response from previous ipc call
with self.retpend_mutex: with self.retpend_mutex:
@@ -96,12 +128,38 @@ class BrokerMp(object):
returns a Queue object which eventually contains the response if want_retval returns a Queue object which eventually contains the response if want_retval
(not-impl here since nothing uses it yet) (not-impl here since nothing uses it yet)
""" """
if dest == "listen": if dest == "httpconn":
for p in self.procs: sck, addr = args
p.q_pend.put([0, dest, [args[0], len(self.procs)]]) sck2 = sck
if PY2:
buf = MemesIO()
ForkingPickler(buf).dump(sck)
sck2 = buf.getvalue()
elif dest == "cb_httpsrv_up": proc = sorted(self.procs, key=lambda x: x.workload)[0]
self.hub.cb_httpsrv_up() proc.q_pend.put([0, dest, [sck2, addr]])
with self.mutex:
proc.clients[addr] = 50
proc.workload += 50
else: else:
raise Exception("what is " + str(dest)) raise Exception("what is " + str(dest))
def debug_load_balancer(self):
fmt = "\033[1m{}\033[0;36m{:4}\033[0m "
if not VT100:
fmt = "({}{:4})"
last = ""
while self.procs:
msg = ""
for proc in self.procs:
msg += fmt.format(len(proc.clients), proc.workload)
if msg != last:
last = msg
with self.hub.log_mutex:
print(msg)
time.sleep(0.1)

View File

@@ -3,13 +3,18 @@ from __future__ import print_function, unicode_literals
from copyparty.authsrv import AuthSrv from copyparty.authsrv import AuthSrv
import sys import sys
import time
import signal import signal
import threading import threading
from .__init__ import PY2, WINDOWS
from .broker_util import ExceptionalQueue from .broker_util import ExceptionalQueue
from .httpsrv import HttpSrv from .httpsrv import HttpSrv
from .util import FAKE_MP from .util import FAKE_MP
if PY2 and not WINDOWS:
import pickle # nosec
class MpWorker(object): class MpWorker(object):
"""one single mp instance""" """one single mp instance"""
@@ -20,23 +25,22 @@ class MpWorker(object):
self.args = args self.args = args
self.n = n self.n = n
self.log = self._log_disabled if args.q and not args.lo else self._log_enabled
self.retpend = {} self.retpend = {}
self.retpend_mutex = threading.Lock() self.retpend_mutex = threading.Lock()
self.mutex = threading.Lock() self.mutex = threading.Lock()
self.workload_thr_alive = False
# we inherited signal_handler from parent, # we inherited signal_handler from parent,
# replace it with something harmless # replace it with something harmless
if not FAKE_MP: if not FAKE_MP:
for sig in [signal.SIGINT, signal.SIGTERM]: signal.signal(signal.SIGINT, self.signal_handler)
signal.signal(sig, self.signal_handler)
# starting to look like a good idea # starting to look like a good idea
self.asrv = AuthSrv(args, None, False) self.asrv = AuthSrv(args, None, False)
# instantiate all services here (TODO: inheritance?) # instantiate all services here (TODO: inheritance?)
self.httpsrv = HttpSrv(self, n) self.httpsrv = HttpSrv(self, True)
self.httpsrv.disconnect_func = self.httpdrop
# on winxp and some other platforms, # on winxp and some other platforms,
# use thr.join() to block all signals # use thr.join() to block all signals
@@ -45,19 +49,19 @@ class MpWorker(object):
thr.start() thr.start()
thr.join() thr.join()
def signal_handler(self, sig, frame): def signal_handler(self, signal, frame):
# print('k') # print('k')
pass pass
def _log_enabled(self, src, msg, c=0): def log(self, src, msg, c=0):
self.q_yield.put([0, "log", [src, msg, c]]) self.q_yield.put([0, "log", [src, msg, c]])
def _log_disabled(self, src, msg, c=0):
pass
def logw(self, msg, c=0): def logw(self, msg, c=0):
self.log("mp{}".format(self.n), msg, c) self.log("mp{}".format(self.n), msg, c)
def httpdrop(self, addr):
self.q_yield.put([0, "httpdrop", [addr]])
def main(self): def main(self):
while True: while True:
retq_id, dest, args = self.q_pend.get() retq_id, dest, args = self.q_pend.get()
@@ -69,8 +73,24 @@ class MpWorker(object):
sys.exit(0) sys.exit(0)
return return
elif dest == "listen": elif dest == "httpconn":
self.httpsrv.listen(args[0], args[1]) sck, addr = args
if PY2:
sck = pickle.loads(sck) # nosec
if self.args.log_conn:
self.log("%s %s" % addr, "|%sC-qpop" % ("-" * 4,), c="1;30")
self.httpsrv.accept(sck, addr)
with self.mutex:
if not self.workload_thr_alive:
self.workload_thr_alive = True
thr = threading.Thread(
target=self.thr_workload, name="mpw-workload"
)
thr.daemon = True
thr.start()
elif dest == "retq": elif dest == "retq":
# response from previous ipc call # response from previous ipc call
@@ -94,3 +114,16 @@ class MpWorker(object):
self.q_yield.put([retq_id, dest, args]) self.q_yield.put([retq_id, dest, args])
return retq return retq
def thr_workload(self):
"""announce workloads to MpSrv (the mp controller / loadbalancer)"""
# avoid locking in extract_filedata by tracking difference here
while True:
time.sleep(0.2)
with self.mutex:
if self.httpsrv.num_clients() == 0:
# no clients rn, termiante thread
self.workload_thr_alive = False
return
self.q_yield.put([0, "workload", [self.httpsrv.workload]])

View File

@@ -3,6 +3,7 @@ from __future__ import print_function, unicode_literals
import threading import threading
from .authsrv import AuthSrv
from .httpsrv import HttpSrv from .httpsrv import HttpSrv
from .broker_util import ExceptionalQueue, try_exec from .broker_util import ExceptionalQueue, try_exec
@@ -17,10 +18,10 @@ class BrokerThr(object):
self.asrv = hub.asrv self.asrv = hub.asrv
self.mutex = threading.Lock() self.mutex = threading.Lock()
self.num_workers = 1
# instantiate all services here (TODO: inheritance?) # instantiate all services here (TODO: inheritance?)
self.httpsrv = HttpSrv(self, None) self.httpsrv = HttpSrv(self)
self.httpsrv.disconnect_func = self.httpdrop
def shutdown(self): def shutdown(self):
# self.log("broker", "shutting down") # self.log("broker", "shutting down")
@@ -28,8 +29,12 @@ class BrokerThr(object):
pass pass
def put(self, want_retval, dest, *args): def put(self, want_retval, dest, *args):
if dest == "listen": if dest == "httpconn":
self.httpsrv.listen(args[0], 1) sck, addr = args
if self.args.log_conn:
self.log("%s %s" % addr, "|%sC-qpop" % ("-" * 4,), c="1;30")
self.httpsrv.accept(sck, addr)
else: else:
# new ipc invoking managed service in hub # new ipc invoking managed service in hub
@@ -46,3 +51,6 @@ class BrokerThr(object):
retq = ExceptionalQueue(1) retq = ExceptionalQueue(1)
retq.put(rv) retq.put(rv)
return retq return retq
def httpdrop(self, addr):
self.hub.tcpsrv.num_clients.add(-1)

View File

@@ -13,12 +13,16 @@ import ctypes
from datetime import datetime from datetime import datetime
import calendar import calendar
from .__init__ import E, PY2, WINDOWS, ANYWIN, unicode from .__init__ import E, PY2, WINDOWS, ANYWIN
from .util import * # noqa # pylint: disable=unused-wildcard-import from .util import * # noqa # pylint: disable=unused-wildcard-import
from .bos import bos
from .authsrv import AuthSrv from .authsrv import AuthSrv
from .szip import StreamZip from .szip import StreamZip
from .star import StreamTar from .star import StreamTar
from .vcr import VCR_Direct
from .th_srv import FMT_FF
if not PY2:
unicode = str
NO_CACHE = {"Cache-Control": "no-cache"} NO_CACHE = {"Cache-Control": "no-cache"}
@@ -38,6 +42,7 @@ class HttpCli(object):
self.ip = conn.addr[0] self.ip = conn.addr[0]
self.addr = conn.addr # type: tuple[str, int] self.addr = conn.addr # type: tuple[str, int]
self.args = conn.args self.args = conn.args
self.is_mp = conn.is_mp
self.asrv = conn.asrv # type: AuthSrv self.asrv = conn.asrv # type: AuthSrv
self.ico = conn.ico self.ico = conn.ico
self.thumbcli = conn.thumbcli self.thumbcli = conn.thumbcli
@@ -59,7 +64,7 @@ class HttpCli(object):
def unpwd(self, m): def unpwd(self, m):
a, b = m.groups() a, b = m.groups()
return "=\033[7m {} \033[27m{}".format(self.asrv.iacct[a], b) return "=\033[7m {} \033[27m{}".format(self.asrv.iuser[a], b)
def _check_nonfatal(self, ex): def _check_nonfatal(self, ex):
return ex.code < 400 or ex.code in [404, 429] return ex.code < 400 or ex.code in [404, 429]
@@ -95,13 +100,9 @@ class HttpCli(object):
try: try:
self.mode, self.req, self.http_ver = headerlines[0].split(" ") self.mode, self.req, self.http_ver = headerlines[0].split(" ")
except: except:
msg = " ]\n#[ ".join(headerlines) raise Pebkac(400, "bad headers:\n" + "\n".join(headerlines))
raise Pebkac(400, "bad headers:\n#[ " + msg + " ]")
except Pebkac as ex: except Pebkac as ex:
self.mode = "GET"
self.req = "[junk]"
self.http_ver = "HTTP/1.1"
# self.log("pebkac at httpcli.run #1: " + repr(ex)) # self.log("pebkac at httpcli.run #1: " + repr(ex))
self.keepalive = self._check_nonfatal(ex) self.keepalive = self._check_nonfatal(ex)
self.loud_reply(unicode(ex), status=ex.code) self.loud_reply(unicode(ex), status=ex.code)
@@ -182,11 +183,9 @@ class HttpCli(object):
self.vpath = unquotep(vpath) self.vpath = unquotep(vpath)
pwd = uparam.get("pw") pwd = uparam.get("pw")
self.uname = self.asrv.iacct.get(pwd, "*") self.uname = self.asrv.iuser.get(pwd, "*")
self.rvol = self.asrv.vfs.aread[self.uname] self.rvol, self.wvol, self.avol = [[], [], []]
self.wvol = self.asrv.vfs.awrite[self.uname] self.asrv.vfs.user_tree(self.uname, self.rvol, self.wvol, self.avol)
self.mvol = self.asrv.vfs.amove[self.uname]
self.dvol = self.asrv.vfs.adel[self.uname]
if pwd and "pw" in self.ouparam and pwd != cookies.get("cppwd"): if pwd and "pw" in self.ouparam and pwd != cookies.get("cppwd"):
self.out_headers["Set-Cookie"] = self.get_pwd_cookie(pwd)[0] self.out_headers["Set-Cookie"] = self.get_pwd_cookie(pwd)[0]
@@ -229,18 +228,19 @@ class HttpCli(object):
except Pebkac: except Pebkac:
return False return False
def send_headers(self, length, status=200, mime=None, headers=None): def send_headers(self, length, status=200, mime=None, headers={}):
response = ["{} {} {}".format(self.http_ver, status, HTTPCODE[status])] response = ["{} {} {}".format(self.http_ver, status, HTTPCODE[status])]
if length is not None: if length is None:
self.keepalive = False
else:
response.append("Content-Length: " + unicode(length)) response.append("Content-Length: " + unicode(length))
# close if unknown length, otherwise take client's preference # close if unknown length, otherwise take client's preference
response.append("Connection: " + ("Keep-Alive" if self.keepalive else "Close")) response.append("Connection: " + ("Keep-Alive" if self.keepalive else "Close"))
# headers{} overrides anything set previously # headers{} overrides anything set previously
if headers: self.out_headers.update(headers)
self.out_headers.update(headers)
# default to utf8 html if no content-type is set # default to utf8 html if no content-type is set
if not mime: if not mime:
@@ -257,7 +257,7 @@ class HttpCli(object):
except: except:
raise Pebkac(400, "client d/c while replying headers") raise Pebkac(400, "client d/c while replying headers")
def reply(self, body, status=200, mime=None, headers=None): def reply(self, body, status=200, mime=None, headers={}):
# TODO something to reply with user-supplied values safely # TODO something to reply with user-supplied values safely
self.send_headers(len(body), status, mime, headers) self.send_headers(len(body), status, mime, headers)
@@ -273,7 +273,7 @@ class HttpCli(object):
self.log(body.rstrip()) self.log(body.rstrip())
self.reply(b"<pre>" + body.encode("utf-8") + b"\r\n", *list(args), **kwargs) self.reply(b"<pre>" + body.encode("utf-8") + b"\r\n", *list(args), **kwargs)
def urlq(self, add, rm): def urlq(self, add={}, rm=[]):
""" """
generates url query based on uparam (b, pw, all others) generates url query based on uparam (b, pw, all others)
removing anything in rm, adding pairs in add removing anything in rm, adding pairs in add
@@ -345,9 +345,6 @@ class HttpCli(object):
if "tree" in self.uparam: if "tree" in self.uparam:
return self.tx_tree() return self.tx_tree()
if "stack" in self.uparam:
return self.tx_stack()
# conditional redirect to single volumes # conditional redirect to single volumes
if self.vpath == "" and not self.ouparam: if self.vpath == "" and not self.ouparam:
nread = len(self.rvol) nread = len(self.rvol)
@@ -362,21 +359,14 @@ class HttpCli(object):
self.redirect(vpath, flavor="redirecting to", use302=True) self.redirect(vpath, flavor="redirecting to", use302=True)
return True return True
x = self.asrv.vfs.can_access(self.vpath, self.uname) self.readable, self.writable = self.asrv.vfs.can_access(self.vpath, self.uname)
self.can_read, self.can_write, self.can_move, self.can_delete = x if not self.readable and not self.writable:
if not self.can_read and not self.can_write:
if self.vpath: if self.vpath:
self.log("inaccessible: [{}]".format(self.vpath)) self.log("inaccessible: [{}]".format(self.vpath))
raise Pebkac(404) raise Pebkac(404)
self.uparam = {"h": False} self.uparam = {"h": False}
if "delete" in self.uparam:
return self.handle_rm()
if "move" in self.uparam:
return self.handle_mv()
if "h" in self.uparam: if "h" in self.uparam:
self.vpath = None self.vpath = None
return self.tx_mounts() return self.tx_mounts()
@@ -384,6 +374,9 @@ class HttpCli(object):
if "scan" in self.uparam: if "scan" in self.uparam:
return self.scanvol() return self.scanvol()
if "stack" in self.uparam:
return self.tx_stack()
return self.tx_browser() return self.tx_browser()
def handle_options(self): def handle_options(self):
@@ -489,17 +482,15 @@ class HttpCli(object):
addr = self.ip.replace(":", ".") addr = self.ip.replace(":", ".")
fn = "put-{:.6f}-{}.bin".format(time.time(), addr) fn = "put-{:.6f}-{}.bin".format(time.time(), addr)
path = os.path.join(fdir, fn) path = os.path.join(fdir, fn)
if self.args.nw:
path = os.devnull
with open(fsenc(path), "wb", 512 * 1024) as f: with open(fsenc(path), "wb", 512 * 1024) as f:
post_sz, _, sha_b64 = hashcopy(reader, f) post_sz, _, sha_b64 = hashcopy(self.conn, reader, f)
if not self.args.nw: vfs, vrem = vfs.get_dbv(rem)
vfs, vrem = vfs.get_dbv(rem)
self.conn.hsrv.broker.put( self.conn.hsrv.broker.put(
False, "up2k.hash_file", vfs.realpath, vfs.flags, vrem, fn False, "up2k.hash_file", vfs.realpath, vfs.flags, vrem, fn
) )
return post_sz, sha_b64, remains, path return post_sz, sha_b64, remains, path
@@ -616,18 +607,17 @@ class HttpCli(object):
if sub: if sub:
try: try:
dst = os.path.join(vfs.realpath, rem) dst = os.path.join(vfs.realpath, rem)
if not bos.path.isdir(dst): if not os.path.isdir(fsenc(dst)):
bos.makedirs(dst) os.makedirs(fsenc(dst))
except OSError as ex: except OSError as ex:
self.log("makedirs failed [{}]".format(dst)) self.log("makedirs failed [{}]".format(dst))
if not bos.path.isdir(dst): if ex.errno == 13:
if ex.errno == 13: raise Pebkac(500, "the server OS denied write-access")
raise Pebkac(500, "the server OS denied write-access")
if ex.errno == 17: if ex.errno == 17:
raise Pebkac(400, "some file got your folder name") raise Pebkac(400, "some file got your folder name")
raise Pebkac(500, min_ex()) raise Pebkac(500, min_ex())
except: except:
raise Pebkac(500, min_ex()) raise Pebkac(500, min_ex())
@@ -725,7 +715,7 @@ class HttpCli(object):
with open(fsenc(path), "rb+", 512 * 1024) as f: with open(fsenc(path), "rb+", 512 * 1024) as f:
f.seek(cstart[0]) f.seek(cstart[0])
post_sz, _, sha_b64 = hashcopy(reader, f) post_sz, _, sha_b64 = hashcopy(self.conn, reader, f)
if sha_b64 != chash: if sha_b64 != chash:
raise Pebkac( raise Pebkac(
@@ -766,7 +756,7 @@ class HttpCli(object):
times = (int(time.time()), int(lastmod)) times = (int(time.time()), int(lastmod))
self.log("no more chunks, setting times {}".format(times)) self.log("no more chunks, setting times {}".format(times))
try: try:
bos.utime(path, times) os.utime(fsenc(path), times)
except: except:
self.log("failed to utime ({}, {})".format(path, times)) self.log("failed to utime ({}, {})".format(path, times))
@@ -785,7 +775,7 @@ class HttpCli(object):
return True return True
def get_pwd_cookie(self, pwd): def get_pwd_cookie(self, pwd):
if pwd in self.asrv.iacct: if pwd in self.asrv.iuser:
msg = "login ok" msg = "login ok"
dt = datetime.utcfromtimestamp(time.time() + 60 * 60 * 24 * 365) dt = datetime.utcfromtimestamp(time.time() + 60 * 60 * 24 * 365)
exp = dt.strftime("%a, %d %b %Y %H:%M:%S GMT") exp = dt.strftime("%a, %d %b %Y %H:%M:%S GMT")
@@ -805,20 +795,20 @@ class HttpCli(object):
vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True) vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
self._assert_safe_rem(rem) self._assert_safe_rem(rem)
sanitized = sanitize_fn(new_dir, "", []) sanitized = sanitize_fn(new_dir)
if not nullwrite: if not nullwrite:
fdir = os.path.join(vfs.realpath, rem) fdir = os.path.join(vfs.realpath, rem)
fn = os.path.join(fdir, sanitized) fn = os.path.join(fdir, sanitized)
if not bos.path.isdir(fdir): if not os.path.isdir(fsenc(fdir)):
raise Pebkac(500, "parent folder does not exist") raise Pebkac(500, "parent folder does not exist")
if bos.path.isdir(fn): if os.path.isdir(fsenc(fn)):
raise Pebkac(500, "that folder exists already") raise Pebkac(500, "that folder exists already")
try: try:
bos.mkdir(fn) os.mkdir(fsenc(fn))
except OSError as ex: except OSError as ex:
if ex.errno == 13: if ex.errno == 13:
raise Pebkac(500, "the server OS denied write-access") raise Pebkac(500, "the server OS denied write-access")
@@ -842,13 +832,13 @@ class HttpCli(object):
if not new_file.endswith(".md"): if not new_file.endswith(".md"):
new_file += ".md" new_file += ".md"
sanitized = sanitize_fn(new_file, "", []) sanitized = sanitize_fn(new_file)
if not nullwrite: if not nullwrite:
fdir = os.path.join(vfs.realpath, rem) fdir = os.path.join(vfs.realpath, rem)
fn = os.path.join(fdir, sanitized) fn = os.path.join(fdir, sanitized)
if bos.path.exists(fn): if os.path.exists(fsenc(fn)):
raise Pebkac(500, "that file exists already") raise Pebkac(500, "that file exists already")
with open(fsenc(fn), "wb") as f: with open(fsenc(fn), "wb") as f:
@@ -875,10 +865,10 @@ class HttpCli(object):
if p_file and not nullwrite: if p_file and not nullwrite:
fdir = os.path.join(vfs.realpath, rem) fdir = os.path.join(vfs.realpath, rem)
fname = sanitize_fn( fname = sanitize_fn(
p_file, "", [".prologue.html", ".epilogue.html"] p_file, bad=[".prologue.html", ".epilogue.html"]
) )
if not bos.path.isdir(fdir): if not os.path.isdir(fsenc(fdir)):
raise Pebkac(404, "that folder does not exist") raise Pebkac(404, "that folder does not exist")
suffix = ".{:.6f}-{}".format(time.time(), self.ip) suffix = ".{:.6f}-{}".format(time.time(), self.ip)
@@ -892,7 +882,7 @@ class HttpCli(object):
with ren_open(fname, "wb", 512 * 1024, **open_args) as f: with ren_open(fname, "wb", 512 * 1024, **open_args) as f:
f, fname = f["orz"] f, fname = f["orz"]
self.log("writing to {}/{}".format(fdir, fname)) self.log("writing to {}/{}".format(fdir, fname))
sz, sha512_hex, _ = hashcopy(p_data, f) sz, sha512_hex, _ = hashcopy(self.conn, p_data, f)
if sz == 0: if sz == 0:
raise Pebkac(400, "empty files in post") raise Pebkac(400, "empty files in post")
@@ -917,10 +907,10 @@ class HttpCli(object):
suffix = ".PARTIAL" suffix = ".PARTIAL"
try: try:
bos.rename(fp, fp2 + suffix) os.rename(fsenc(fp), fsenc(fp2 + suffix))
except: except:
fp2 = fp2[: -len(suffix) - 1] fp2 = fp2[: -len(suffix) - 1]
bos.rename(fp, fp2 + suffix) os.rename(fsenc(fp), fsenc(fp2 + suffix))
raise raise
@@ -1004,6 +994,13 @@ class HttpCli(object):
vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True) vfs, rem = self.asrv.vfs.get(self.vpath, self.uname, False, True)
self._assert_safe_rem(rem) self._assert_safe_rem(rem)
# TODO:
# the per-volume read/write permissions must be replaced with permission flags
# which would decide how to handle uploads to filenames which are taken,
# current behavior of creating a new name is a good default for binary files
# but should also offer a flag to takeover the filename and rename the old one
#
# stopgap:
if not rem.endswith(".md"): if not rem.endswith(".md"):
raise Pebkac(400, "only markdown pls") raise Pebkac(400, "only markdown pls")
@@ -1018,7 +1015,7 @@ class HttpCli(object):
fp = os.path.join(vfs.realpath, rem) fp = os.path.join(vfs.realpath, rem)
srv_lastmod = srv_lastmod3 = -1 srv_lastmod = srv_lastmod3 = -1
try: try:
st = bos.stat(fp) st = os.stat(fsenc(fp))
srv_lastmod = st.st_mtime srv_lastmod = st.st_mtime
srv_lastmod3 = int(srv_lastmod * 1000) srv_lastmod3 = int(srv_lastmod * 1000)
except OSError as ex: except OSError as ex:
@@ -1054,22 +1051,23 @@ class HttpCli(object):
self.reply(response.encode("utf-8")) self.reply(response.encode("utf-8"))
return True return True
# TODO another hack re: pending permissions rework
mdir, mfile = os.path.split(fp) mdir, mfile = os.path.split(fp)
mfile2 = "{}.{:.3f}.md".format(mfile[:-3], srv_lastmod) mfile2 = "{}.{:.3f}.md".format(mfile[:-3], srv_lastmod)
try: try:
bos.mkdir(os.path.join(mdir, ".hist")) os.mkdir(fsenc(os.path.join(mdir, ".hist")))
except: except:
pass pass
bos.rename(fp, os.path.join(mdir, ".hist", mfile2)) os.rename(fsenc(fp), fsenc(os.path.join(mdir, ".hist", mfile2)))
p_field, _, p_data = next(self.parser.gen) p_field, _, p_data = next(self.parser.gen)
if p_field != "body": if p_field != "body":
raise Pebkac(400, "expected body, got {}".format(p_field)) raise Pebkac(400, "expected body, got {}".format(p_field))
with open(fsenc(fp), "wb", 512 * 1024) as f: with open(fsenc(fp), "wb", 512 * 1024) as f:
sz, sha512, _ = hashcopy(p_data, f) sz, sha512, _ = hashcopy(self.conn, p_data, f)
new_lastmod = bos.stat(fp).st_mtime new_lastmod = os.stat(fsenc(fp)).st_mtime
new_lastmod3 = int(new_lastmod * 1000) new_lastmod3 = int(new_lastmod * 1000)
sha512 = sha512[:56] sha512 = sha512[:56]
@@ -1114,7 +1112,7 @@ class HttpCli(object):
for ext in ["", ".gz", ".br"]: for ext in ["", ".gz", ".br"]:
try: try:
fs_path = req_path + ext fs_path = req_path + ext
st = bos.stat(fs_path) st = os.stat(fsenc(fs_path))
file_ts = max(file_ts, st.st_mtime) file_ts = max(file_ts, st.st_mtime)
editions[ext or "plain"] = [fs_path, st.st_size] editions[ext or "plain"] = [fs_path, st.st_size]
except: except:
@@ -1257,7 +1255,8 @@ class HttpCli(object):
if use_sendfile: if use_sendfile:
remains = sendfile_kern(lower, upper, f, self.s) remains = sendfile_kern(lower, upper, f, self.s)
else: else:
remains = sendfile_py(lower, upper, f, self.s) actor = self.conn if self.is_mp else None
remains = sendfile_py(lower, upper, f, self.s, actor)
if remains > 0: if remains > 0:
logmsg += " \033[31m" + unicode(upper - remains) + "\033[0m" logmsg += " \033[31m" + unicode(upper - remains) + "\033[0m"
@@ -1314,7 +1313,7 @@ class HttpCli(object):
fgen = vn.zipgen(rem, items, self.uname, dots, not self.args.no_scandir) fgen = vn.zipgen(rem, items, self.uname, dots, not self.args.no_scandir)
# for f in fgen: print(repr({k: f[k] for k in ["vp", "ap"]})) # for f in fgen: print(repr({k: f[k] for k in ["vp", "ap"]}))
bgen = packer(self.log, fgen, utf8="utf" in uarg, pre_crc="crc" in uarg) bgen = packer(fgen, utf8="utf" in uarg, pre_crc="crc" in uarg)
bsent = 0 bsent = 0
for buf in bgen.gen(): for buf in bgen.gen():
if not buf: if not buf:
@@ -1366,10 +1365,10 @@ class HttpCli(object):
html_path = os.path.join(E.mod, "web", "{}.html".format(tpl)) html_path = os.path.join(E.mod, "web", "{}.html".format(tpl))
template = self.j2(tpl) template = self.j2(tpl)
st = bos.stat(fs_path) st = os.stat(fsenc(fs_path))
ts_md = st.st_mtime ts_md = st.st_mtime
st = bos.stat(html_path) st = os.stat(fsenc(html_path))
ts_html = st.st_mtime ts_html = st.st_mtime
sz_md = 0 sz_md = 0
@@ -1378,7 +1377,7 @@ class HttpCli(object):
for c, v in [[b"&", 4], [b"<", 3], [b">", 3]]: for c, v in [[b"&", 4], [b"<", 3], [b">", 3]]:
sz_md += (len(buf) - len(buf.replace(c, b""))) * v sz_md += (len(buf) - len(buf.replace(c, b""))) * v
file_ts = max(ts_md, ts_html, E.t0) file_ts = max(ts_md, ts_html)
file_lastmod, do_send = self._chk_lastmod(file_ts) file_lastmod, do_send = self._chk_lastmod(file_ts)
self.out_headers["Last-Modified"] = file_lastmod self.out_headers["Last-Modified"] = file_lastmod
self.out_headers.update(NO_CACHE) self.out_headers.update(NO_CACHE)
@@ -1425,14 +1424,13 @@ class HttpCli(object):
return True return True
def tx_mounts(self): def tx_mounts(self):
suf = self.urlq({}, ["h"]) suf = self.urlq(rm=["h"])
avol = [x for x in self.wvol if x in self.rvol]
rvol, wvol, avol = [ rvol, wvol, avol = [
[("/" + x).rstrip("/") + "/" for x in y] [("/" + x).rstrip("/") + "/" for x in y]
for y in [self.rvol, self.wvol, avol] for y in [self.rvol, self.wvol, self.avol]
] ]
if avol and not self.args.no_rescan: if self.avol and not self.args.no_rescan:
x = self.conn.hsrv.broker.put(True, "up2k.get_state") x = self.conn.hsrv.broker.put(True, "up2k.get_state")
vs = json.loads(x.get()) vs = json.loads(x.get())
vstate = {("/" + k).rstrip("/") + "/": v for k, v in vs["volstate"].items()} vstate = {("/" + k).rstrip("/") + "/": v for k, v in vs["volstate"].items()}
@@ -1457,8 +1455,8 @@ class HttpCli(object):
return True return True
def scanvol(self): def scanvol(self):
if not self.can_read or not self.can_write: if not self.readable or not self.writable:
raise Pebkac(403, "not allowed for user " + self.uname) raise Pebkac(403, "not admin")
if self.args.no_rescan: if self.args.no_rescan:
raise Pebkac(403, "disabled by argv") raise Pebkac(403, "disabled by argv")
@@ -1476,8 +1474,8 @@ class HttpCli(object):
raise Pebkac(500, x) raise Pebkac(500, x)
def tx_stack(self): def tx_stack(self):
if not [x for x in self.wvol if x in self.rvol]: if not self.readable or not self.writable:
raise Pebkac(403, "not allowed for user " + self.uname) raise Pebkac(403, "not admin")
if self.args.no_stack: if self.args.no_stack:
raise Pebkac(403, "disabled by argv") raise Pebkac(403, "disabled by argv")
@@ -1515,7 +1513,7 @@ class HttpCli(object):
try: try:
vn, rem = self.asrv.vfs.get(top, self.uname, True, False) vn, rem = self.asrv.vfs.get(top, self.uname, True, False)
fsroot, vfs_ls, vfs_virt = vn.ls( fsroot, vfs_ls, vfs_virt = vn.ls(
rem, self.uname, not self.args.no_scandir, [[True], [False, True]] rem, self.uname, not self.args.no_scandir, incl_wo=True
) )
except: except:
vfs_ls = [] vfs_ls = []
@@ -1542,33 +1540,6 @@ class HttpCli(object):
ret["a"] = dirs ret["a"] = dirs
return ret return ret
def handle_rm(self):
if not self.can_delete:
raise Pebkac(403, "not allowed for user " + self.uname)
if self.args.no_del:
raise Pebkac(403, "disabled by argv")
x = self.conn.hsrv.broker.put(True, "up2k.handle_rm", self.uname, self.vpath)
self.loud_reply(x.get())
def handle_mv(self):
if not self.can_move:
raise Pebkac(403, "not allowed for user " + self.uname)
if self.args.no_mv:
raise Pebkac(403, "disabled by argv")
# full path of new loc (incl filename)
dst = self.uparam.get("move")
if not dst:
raise Pebkac(400, "need dst vpath")
x = self.conn.hsrv.broker.put(
True, "up2k.handle_mv", self.uname, self.vpath, dst
)
self.loud_reply(x.get())
def tx_browser(self): def tx_browser(self):
vpath = "" vpath = ""
vpnodes = [["", "/"]] vpnodes = [["", "/"]]
@@ -1581,28 +1552,37 @@ class HttpCli(object):
vpnodes.append([quotep(vpath) + "/", html_escape(node, crlf=True)]) vpnodes.append([quotep(vpath) + "/", html_escape(node, crlf=True)])
vn, rem = self.asrv.vfs.get(self.vpath, self.uname, False, False) vn, rem = self.asrv.vfs.get(
self.vpath, self.uname, self.readable, self.writable
)
abspath = vn.canonical(rem) abspath = vn.canonical(rem)
dbv, vrem = vn.get_dbv(rem) dbv, vrem = vn.get_dbv(rem)
try: try:
st = bos.stat(abspath) st = os.stat(fsenc(abspath))
except: except:
raise Pebkac(404) raise Pebkac(404)
if self.can_read: if self.readable:
if rem.startswith(".hist/up2k.") or ( if rem.startswith(".hist/up2k."):
rem.endswith("/dir.txt") and rem.startswith(".hist/th/")
):
raise Pebkac(403) raise Pebkac(403)
if "vcr" in self.uparam:
ext = abspath.rsplit(".")[-1]
if not self.args.vcr or ext not in FMT_FF:
raise Pebkac(403)
vcr = VCR_Direct(self, abspath)
vcr.run()
return False
is_dir = stat.S_ISDIR(st.st_mode) is_dir = stat.S_ISDIR(st.st_mode)
th_fmt = self.uparam.get("th") th_fmt = self.uparam.get("th")
if th_fmt is not None: if th_fmt is not None:
if is_dir: if is_dir:
for fn in self.args.th_covers.split(","): for fn in self.args.th_covers.split(","):
fp = os.path.join(abspath, fn) fp = os.path.join(abspath, fn)
if bos.path.exists(fp): if os.path.exists(fp):
vrem = "{}/{}".format(vrem.rstrip("/"), fn) vrem = "{}/{}".format(vrem.rstrip("/"), fn)
is_dir = False is_dir = False
break break
@@ -1657,16 +1637,12 @@ class HttpCli(object):
srv_info = "</span> /// <span>".join(srv_info) srv_info = "</span> /// <span>".join(srv_info)
perms = [] perms = []
if self.can_read: if self.readable:
perms.append("read") perms.append("read")
if self.can_write: if self.writable:
perms.append("write") perms.append("write")
if self.can_move:
perms.append("move")
if self.can_delete:
perms.append("delete")
url_suf = self.urlq({}, []) url_suf = self.urlq()
is_ls = "ls" in self.uparam is_ls = "ls" in self.uparam
tpl = "browser" tpl = "browser"
@@ -1676,7 +1652,7 @@ class HttpCli(object):
logues = ["", ""] logues = ["", ""]
for n, fn in enumerate([".prologue.html", ".epilogue.html"]): for n, fn in enumerate([".prologue.html", ".epilogue.html"]):
fn = os.path.join(abspath, fn) fn = os.path.join(abspath, fn)
if bos.path.exists(fn): if os.path.exists(fsenc(fn)):
with open(fsenc(fn), "rb") as f: with open(fsenc(fn), "rb") as f:
logues[n] = f.read().decode("utf-8") logues[n] = f.read().decode("utf-8")
@@ -1685,7 +1661,6 @@ class HttpCli(object):
"files": [], "files": [],
"taglist": [], "taglist": [],
"srvinf": srv_info, "srvinf": srv_info,
"acct": self.uname,
"perms": perms, "perms": perms,
"logues": logues, "logues": logues,
} }
@@ -1693,22 +1668,19 @@ class HttpCli(object):
"vdir": quotep(self.vpath), "vdir": quotep(self.vpath),
"vpnodes": vpnodes, "vpnodes": vpnodes,
"files": [], "files": [],
"acct": self.uname,
"perms": json.dumps(perms), "perms": json.dumps(perms),
"taglist": [], "taglist": [],
"tag_order": [], "tag_order": [],
"have_up2k_idx": ("e2d" in vn.flags), "have_up2k_idx": ("e2d" in vn.flags),
"have_tags_idx": ("e2t" in vn.flags), "have_tags_idx": ("e2t" in vn.flags),
"have_mv": (not self.args.no_mv),
"have_del": (not self.args.no_del),
"have_zip": (not self.args.no_zip), "have_zip": (not self.args.no_zip),
"have_b_u": (self.can_write and self.uparam.get("b") == "u"), "have_b_u": (self.writable and self.uparam.get("b") == "u"),
"url_suf": url_suf, "url_suf": url_suf,
"logues": logues, "logues": logues,
"title": html_escape(self.vpath, crlf=True), "title": html_escape(self.vpath, crlf=True),
"srv_info": srv_info, "srv_info": srv_info,
} }
if not self.can_read: if not self.readable:
if is_ls: if is_ls:
ret = json.dumps(ls_ret) ret = json.dumps(ls_ret)
self.reply( self.reply(
@@ -1731,7 +1703,7 @@ class HttpCli(object):
return self.tx_zip(k, v, vn, rem, [], self.args.ed) return self.tx_zip(k, v, vn, rem, [], self.args.ed)
fsroot, vfs_ls, vfs_virt = vn.ls( fsroot, vfs_ls, vfs_virt = vn.ls(
rem, self.uname, not self.args.no_scandir, [[True], [False, True]] rem, self.uname, not self.args.no_scandir, incl_wo=True
) )
stats = {k: v for k, v in vfs_ls} stats = {k: v for k, v in vfs_ls}
vfs_ls = [x[0] for x in vfs_ls] vfs_ls = [x[0] for x in vfs_ls]
@@ -1742,7 +1714,7 @@ class HttpCli(object):
histdir = os.path.join(fsroot, ".hist") histdir = os.path.join(fsroot, ".hist")
ptn = re.compile(r"(.*)\.([0-9]+\.[0-9]{3})(\.[^\.]+)$") ptn = re.compile(r"(.*)\.([0-9]+\.[0-9]{3})(\.[^\.]+)$")
try: try:
for hfn in bos.listdir(histdir): for hfn in os.listdir(histdir):
m = ptn.match(hfn) m = ptn.match(hfn)
if not m: if not m:
continue continue
@@ -1783,7 +1755,7 @@ class HttpCli(object):
fspath = fsroot + "/" + fn fspath = fsroot + "/" + fn
try: try:
inf = stats.get(fn) or bos.stat(fspath) inf = stats.get(fn) or os.stat(fsenc(fspath))
except: except:
self.log("broken symlink: {}".format(repr(fspath))) self.log("broken symlink: {}".format(repr(fspath)))
continue continue

View File

@@ -34,6 +34,7 @@ class HttpConn(object):
self.args = hsrv.args self.args = hsrv.args
self.asrv = hsrv.asrv self.asrv = hsrv.asrv
self.is_mp = hsrv.is_mp
self.cert_path = hsrv.cert_path self.cert_path = hsrv.cert_path
enth = HAVE_PIL and not self.args.no_thumb enth = HAVE_PIL and not self.args.no_thumb
@@ -44,6 +45,7 @@ class HttpConn(object):
self.stopping = False self.stopping = False
self.nreq = 0 self.nreq = 0
self.nbyte = 0 self.nbyte = 0
self.workload = 0
self.u2idx = None self.u2idx = None
self.log_func = hsrv.log self.log_func = hsrv.log
self.lf_url = re.compile(self.args.lf_url) if self.args.lf_url else None self.lf_url = re.compile(self.args.lf_url) if self.args.lf_url else None
@@ -182,6 +184,11 @@ class HttpConn(object):
self.sr = Unrecv(self.s) self.sr = Unrecv(self.s)
while not self.stopping: while not self.stopping:
if self.is_mp:
self.workload += 50
if self.workload >= 2 ** 31:
self.workload = 100
self.nreq += 1 self.nreq += 1
cli = HttpCli(self) cli = HttpCli(self)
if not cli.run(): if not cli.run():

View File

@@ -4,8 +4,8 @@ from __future__ import print_function, unicode_literals
import os import os
import sys import sys
import time import time
import math
import base64 import base64
import struct
import socket import socket
import threading import threading
@@ -26,16 +26,9 @@ except ImportError:
) )
sys.exit(1) sys.exit(1)
from .__init__ import E, PY2, MACOS from .__init__ import E, MACOS
from .util import spack, min_ex, start_stackmon, start_log_thrs
from .bos import bos
from .httpconn import HttpConn from .httpconn import HttpConn
if PY2:
import Queue as queue
else:
import queue
class HttpSrv(object): class HttpSrv(object):
""" """
@@ -43,26 +36,19 @@ class HttpSrv(object):
relying on MpSrv for performance (HttpSrv is just plain threads) relying on MpSrv for performance (HttpSrv is just plain threads)
""" """
def __init__(self, broker, nid): def __init__(self, broker, is_mp=False):
self.broker = broker self.broker = broker
self.nid = nid self.is_mp = is_mp
self.args = broker.args self.args = broker.args
self.log = broker.log self.log = broker.log
self.asrv = broker.asrv self.asrv = broker.asrv
self.name = "httpsrv" + ("-n{}-i{:x}".format(nid, os.getpid()) if nid else "") self.disconnect_func = None
self.mutex = threading.Lock() self.mutex = threading.Lock()
self.stopping = False
self.tp_nthr = 0 # actual self.clients = {}
self.tp_ncli = 0 # fading self.workload = 0
self.tp_time = None # latest worker collect self.workload_thr_alive = False
self.tp_q = None if self.args.no_htp else queue.LifoQueue()
self.srvs = []
self.ncli = 0 # exact
self.clients = {} # laggy
self.nclimax = 0
self.cb_ts = 0 self.cb_ts = 0
self.cb_v = 0 self.cb_v = 0
@@ -74,161 +60,29 @@ class HttpSrv(object):
} }
cert_path = os.path.join(E.cfg, "cert.pem") cert_path = os.path.join(E.cfg, "cert.pem")
if bos.path.exists(cert_path): if os.path.exists(cert_path):
self.cert_path = cert_path self.cert_path = cert_path
else: else:
self.cert_path = None self.cert_path = None
if self.tp_q:
self.start_threads(4)
name = "httpsrv-scaler" + ("-{}".format(nid) if nid else "")
t = threading.Thread(target=self.thr_scaler, name=name)
t.daemon = True
t.start()
if nid:
if self.args.stackmon:
start_stackmon(self.args.stackmon, nid)
if self.args.log_thrs:
start_log_thrs(self.log, self.args.log_thrs, nid)
def start_threads(self, n):
self.tp_nthr += n
if self.args.log_htp:
self.log(self.name, "workers += {} = {}".format(n, self.tp_nthr), 6)
for _ in range(n):
thr = threading.Thread(
target=self.thr_poolw,
name=self.name + "-poolw",
)
thr.daemon = True
thr.start()
def stop_threads(self, n):
self.tp_nthr -= n
if self.args.log_htp:
self.log(self.name, "workers -= {} = {}".format(n, self.tp_nthr), 6)
for _ in range(n):
self.tp_q.put(None)
def thr_scaler(self):
while True:
time.sleep(2 if self.tp_ncli else 30)
with self.mutex:
self.tp_ncli = max(self.ncli, self.tp_ncli - 2)
if self.tp_nthr > self.tp_ncli + 8:
self.stop_threads(4)
def listen(self, sck, nlisteners):
ip, port = sck.getsockname()
self.srvs.append(sck)
self.nclimax = math.ceil(self.args.nc * 1.0 / nlisteners)
t = threading.Thread(
target=self.thr_listen,
args=(sck,),
name="httpsrv-n{}-listen-{}-{}".format(self.nid or "0", ip, port),
)
t.daemon = True
t.start()
def thr_listen(self, srv_sck):
"""listens on a shared tcp server"""
ip, port = srv_sck.getsockname()
fno = srv_sck.fileno()
msg = "subscribed @ {}:{} f{}".format(ip, port, fno)
self.log(self.name, msg)
self.broker.put(False, "cb_httpsrv_up")
while not self.stopping:
if self.args.log_conn:
self.log(self.name, "|%sC-ncli" % ("-" * 1,), c="1;30")
if self.ncli >= self.nclimax:
self.log(self.name, "at connection limit; waiting", 3)
while self.ncli >= self.nclimax:
time.sleep(0.1)
if self.args.log_conn:
self.log(self.name, "|%sC-acc1" % ("-" * 2,), c="1;30")
try:
sck, addr = srv_sck.accept()
except (OSError, socket.error) as ex:
self.log(self.name, "accept({}): {}".format(fno, ex), c=6)
time.sleep(0.02)
continue
if self.args.log_conn:
m = "|{}C-acc2 \033[0;36m{} \033[3{}m{}".format(
"-" * 3, ip, port % 8, port
)
self.log("%s %s" % addr, m, c="1;30")
self.accept(sck, addr)
def accept(self, sck, addr): def accept(self, sck, addr):
"""takes an incoming tcp connection and creates a thread to handle it""" """takes an incoming tcp connection and creates a thread to handle it"""
now = time.time() if self.args.log_conn:
self.log("%s %s" % addr, "|%sC-cthr" % ("-" * 5,), c="1;30")
if now - (self.tp_time or now) > 300:
self.tp_q = None
if self.tp_q:
self.tp_q.put((sck, addr))
with self.mutex:
self.ncli += 1
self.tp_time = self.tp_time or now
self.tp_ncli = max(self.tp_ncli, self.ncli + 1)
if self.tp_nthr < self.ncli + 4:
self.start_threads(8)
return
if not self.args.no_htp:
m = "looks like the httpserver threadpool died; please make an issue on github and tell me the story of how you pulled that off, thanks and dog bless\n"
self.log(self.name, m, 1)
with self.mutex:
self.ncli += 1
thr = threading.Thread( thr = threading.Thread(
target=self.thr_client, target=self.thr_client,
args=(sck, addr), args=(sck, addr),
name="httpconn-{}-{}".format(addr[0].split(".", 2)[-1][-6:], addr[1]), name="httpsrv-{}-{}".format(addr[0].split(".", 2)[-1][-6:], addr[1]),
) )
thr.daemon = True thr.daemon = True
thr.start() thr.start()
def thr_poolw(self): def num_clients(self):
while True: with self.mutex:
task = self.tp_q.get() return len(self.clients)
if not task:
break
with self.mutex:
self.tp_time = None
try:
sck, addr = task
me = threading.current_thread()
me.name = "httpconn-{}-{}".format(
addr[0].split(".", 2)[-1][-6:], addr[1]
)
self.thr_client(sck, addr)
me.name = self.name + "-poolw"
except:
self.log(self.name, "thr_client: " + min_ex(), 3)
def shutdown(self): def shutdown(self):
self.stopping = True
for srv in self.srvs:
try:
srv.close()
except:
pass
clients = list(self.clients.keys()) clients = list(self.clients.keys())
for cli in clients: for cli in clients:
try: try:
@@ -236,14 +90,7 @@ class HttpSrv(object):
except: except:
pass pass
if self.tp_q: self.log("httpsrv-n", "ok bye")
self.stop_threads(self.tp_nthr)
for _ in range(10):
time.sleep(0.05)
if self.tp_q.empty():
break
self.log(self.name, "ok bye")
def thr_client(self, sck, addr): def thr_client(self, sck, addr):
"""thread managing one tcp client""" """thread managing one tcp client"""
@@ -253,15 +100,25 @@ class HttpSrv(object):
with self.mutex: with self.mutex:
self.clients[cli] = 0 self.clients[cli] = 0
if self.is_mp:
self.workload += 50
if not self.workload_thr_alive:
self.workload_thr_alive = True
thr = threading.Thread(
target=self.thr_workload, name="httpsrv-workload"
)
thr.daemon = True
thr.start()
fno = sck.fileno() fno = sck.fileno()
try: try:
if self.args.log_conn: if self.args.log_conn:
self.log("%s %s" % addr, "|%sC-crun" % ("-" * 4,), c="1;30") self.log("%s %s" % addr, "|%sC-crun" % ("-" * 6,), c="1;30")
cli.run() cli.run()
except (OSError, socket.error) as ex: except (OSError, socket.error) as ex:
if ex.errno not in [10038, 10054, 107, 57, 49, 9]: if ex.errno not in [10038, 10054, 107, 57, 9]:
self.log( self.log(
"%s %s" % addr, "%s %s" % addr,
"run({}): {}".format(fno, ex), "run({}): {}".format(fno, ex),
@@ -271,7 +128,7 @@ class HttpSrv(object):
finally: finally:
sck = cli.s sck = cli.s
if self.args.log_conn: if self.args.log_conn:
self.log("%s %s" % addr, "|%sC-cdone" % ("-" * 5,), c="1;30") self.log("%s %s" % addr, "|%sC-cdone" % ("-" * 7,), c="1;30")
try: try:
fno = sck.fileno() fno = sck.fileno()
@@ -295,7 +152,35 @@ class HttpSrv(object):
finally: finally:
with self.mutex: with self.mutex:
del self.clients[cli] del self.clients[cli]
self.ncli -= 1
if self.disconnect_func:
self.disconnect_func(addr) # pylint: disable=not-callable
def thr_workload(self):
"""indicates the python interpreter workload caused by this HttpSrv"""
# avoid locking in extract_filedata by tracking difference here
while True:
time.sleep(0.2)
with self.mutex:
if not self.clients:
# no clients rn, termiante thread
self.workload_thr_alive = False
self.workload = 0
return
total = 0
with self.mutex:
for cli in self.clients.keys():
now = cli.workload
delta = now - self.clients[cli]
if delta < 0:
# was reset in HttpCli to prevent overflow
delta = now
total += delta
self.clients[cli] = now
self.workload = total
def cachebuster(self): def cachebuster(self):
if time.time() - self.cb_ts < 1: if time.time() - self.cb_ts < 1:
@@ -309,12 +194,12 @@ class HttpSrv(object):
try: try:
with os.scandir(os.path.join(E.mod, "web")) as dh: with os.scandir(os.path.join(E.mod, "web")) as dh:
for fh in dh: for fh in dh:
inf = fh.stat() inf = fh.stat(follow_symlinks=False)
v = max(v, inf.st_mtime) v = max(v, inf.st_mtime)
except: except:
pass pass
v = base64.urlsafe_b64encode(spack(b">xxL", int(v))) v = base64.urlsafe_b64encode(struct.pack(">xxL", int(v)))
self.cb_v = v.decode("ascii")[-4:] self.cb_v = v.decode("ascii")[-4:]
self.cb_ts = time.time() self.cb_ts = time.time()
return self.cb_v return self.cb_v

View File

@@ -7,9 +7,11 @@ import json
import shutil import shutil
import subprocess as sp import subprocess as sp
from .__init__ import PY2, WINDOWS, unicode from .__init__ import PY2, WINDOWS
from .util import fsenc, fsdec, uncyg, REKOBO_LKEY from .util import fsenc, fsdec, uncyg, REKOBO_LKEY
from .bos import bos
if not PY2:
unicode = str
def have_ff(cmd): def have_ff(cmd):
@@ -45,7 +47,7 @@ class MParser(object):
if WINDOWS: if WINDOWS:
bp = uncyg(bp) bp = uncyg(bp)
if bos.path.exists(bp): if os.path.exists(bp):
self.bin = bp self.bin = bp
return return
except: except:
@@ -228,47 +230,37 @@ def parse_ffprobe(txt):
class MTag(object): class MTag(object):
def __init__(self, log_func, args): def __init__(self, log_func, args):
self.log_func = log_func self.log_func = log_func
self.args = args
self.usable = True self.usable = True
self.prefer_mt = not args.no_mtag_ff self.prefer_mt = False
self.backend = "ffprobe" if args.no_mutagen else "mutagen"
self.can_ffprobe = (
HAVE_FFPROBE
and not args.no_mtag_ff
and (not WINDOWS or sys.version_info >= (3, 8))
)
mappings = args.mtm mappings = args.mtm
or_ffprobe = " or FFprobe" self.backend = "ffprobe" if args.no_mutagen else "mutagen"
or_ffprobe = " or ffprobe"
if self.backend == "mutagen": if self.backend == "mutagen":
self.get = self.get_mutagen self.get = self.get_mutagen
try: try:
import mutagen import mutagen
except: except:
self.log("could not load Mutagen, trying FFprobe instead", c=3) self.log("could not load mutagen, trying ffprobe instead", c=3)
self.backend = "ffprobe" self.backend = "ffprobe"
if self.backend == "ffprobe": if self.backend == "ffprobe":
self.usable = self.can_ffprobe
self.get = self.get_ffprobe self.get = self.get_ffprobe
self.prefer_mt = True self.prefer_mt = True
# about 20x slower
self.usable = HAVE_FFPROBE
if not HAVE_FFPROBE: if self.usable and WINDOWS and sys.version_info < (3, 8):
pass self.usable = False
elif args.no_mtag_ff:
msg = "found FFprobe but it was disabled by --no-mtag-ff"
self.log(msg, c=3)
elif WINDOWS and sys.version_info < (3, 8):
or_ffprobe = " or python >= 3.8" or_ffprobe = " or python >= 3.8"
msg = "found FFprobe but your python is too old; need 3.8 or newer" msg = "found ffprobe but your python is too old; need 3.8 or newer"
self.log(msg, c=1) self.log(msg, c=1)
if not self.usable: if not self.usable:
msg = "need Mutagen{} to read media tags so please run this:\n{}{} -m pip install --user mutagen\n" msg = "need mutagen{} to read media tags so please run this:\n{}{} -m pip install --user mutagen\n"
pybin = os.path.basename(sys.executable) self.log(
self.log(msg.format(or_ffprobe, " " * 37, pybin), c=1) msg.format(or_ffprobe, " " * 37, os.path.basename(sys.executable)), c=1
)
return return
# https://picard-docs.musicbrainz.org/downloads/MusicBrainz_Picard_Tag_Map.html # https://picard-docs.musicbrainz.org/downloads/MusicBrainz_Picard_Tag_Map.html
@@ -398,7 +390,7 @@ class MTag(object):
v2 = r2.get(k) v2 = r2.get(k)
if v1 == v2: if v1 == v2:
print(" ", k, v1) print(" ", k, v1)
elif v1 != "0000": # FFprobe date=0 elif v1 != "0000": # ffprobe date=0
diffs.append(k) diffs.append(k)
print(" 1", k, v1) print(" 1", k, v1)
print(" 2", k, v2) print(" 2", k, v2)
@@ -419,33 +411,20 @@ class MTag(object):
md = mutagen.File(fsenc(abspath), easy=True) md = mutagen.File(fsenc(abspath), easy=True)
x = md.info.length x = md.info.length
except Exception as ex: except Exception as ex:
return self.get_ffprobe(abspath) if self.can_ffprobe else {} return {}
sz = bos.path.getsize(abspath) ret = {}
ret = {".q": [0, int((sz / md.info.length) / 128)]} try:
dur = int(md.info.length)
for attr, k, norm in [
["codec", "ac", unicode],
["channels", "chs", int],
["sample_rate", ".hz", int],
["bitrate", ".aq", int],
["length", ".dur", int],
]:
try: try:
v = getattr(md.info, attr) q = int(md.info.bitrate / 1024)
except: except:
continue q = int((os.path.getsize(fsenc(abspath)) / dur) / 128)
if not v: ret[".dur"] = [0, dur]
continue ret[".q"] = [0, q]
except:
if k == ".aq": pass
v /= 1000
if k == "ac" and v.startswith("mp4a.40."):
v = "aac"
ret[k] = [0, norm(v)]
return self.normalize_tags(ret, md) return self.normalize_tags(ret, md)

View File

@@ -1,12 +1,12 @@
# coding: utf-8 # coding: utf-8
from __future__ import print_function, unicode_literals from __future__ import print_function, unicode_literals
import os
import tarfile import tarfile
import threading import threading
from .sutil import errdesc from .sutil import errdesc
from .util import Queue, fsenc from .util import Queue, fsenc
from .bos import bos
class QFile(object): class QFile(object):
@@ -33,11 +33,10 @@ class QFile(object):
class StreamTar(object): class StreamTar(object):
"""construct in-memory tar file from the given path""" """construct in-memory tar file from the given path"""
def __init__(self, log, fgen, **kwargs): def __init__(self, fgen, **kwargs):
self.ci = 0 self.ci = 0
self.co = 0 self.co = 0
self.qfile = QFile() self.qfile = QFile()
self.log = log
self.fgen = fgen self.fgen = fgen
self.errf = None self.errf = None
@@ -61,7 +60,7 @@ class StreamTar(object):
yield None yield None
if self.errf: if self.errf:
bos.unlink(self.errf["ap"]) os.unlink(self.errf["ap"])
def ser(self, f): def ser(self, f):
name = f["vp"] name = f["vp"]
@@ -92,8 +91,7 @@ class StreamTar(object):
errors.append([f["vp"], repr(ex)]) errors.append([f["vp"], repr(ex)])
if errors: if errors:
self.errf, txt = errdesc(errors) self.errf = errdesc(errors)
self.log("\n".join(([repr(self.errf)] + txt[1:])))
self.ser(self.errf) self.ser(self.errf)
self.tar.close() self.tar.close()

View File

@@ -1,12 +1,11 @@
# coding: utf-8 # coding: utf-8
from __future__ import print_function, unicode_literals from __future__ import print_function, unicode_literals
import os
import time import time
import tempfile import tempfile
from datetime import datetime from datetime import datetime
from .bos import bos
def errdesc(errors): def errdesc(errors):
report = ["copyparty failed to add the following files to the archive:", ""] report = ["copyparty failed to add the following files to the archive:", ""]
@@ -21,9 +20,9 @@ def errdesc(errors):
dt = datetime.utcfromtimestamp(time.time()) dt = datetime.utcfromtimestamp(time.time())
dt = dt.strftime("%Y-%m%d-%H%M%S") dt = dt.strftime("%Y-%m%d-%H%M%S")
bos.chmod(tf_path, 0o444) os.chmod(tf_path, 0o444)
return { return {
"vp": "archive-errors-{}.txt".format(dt), "vp": "archive-errors-{}.txt".format(dt),
"ap": tf_path, "ap": tf_path,
"st": bos.stat(tf_path), "st": os.stat(tf_path),
}, report }

View File

@@ -5,16 +5,12 @@ import re
import os import os
import sys import sys
import time import time
import shlex
import string
import signal
import socket
import threading import threading
from datetime import datetime, timedelta from datetime import datetime, timedelta
import calendar import calendar
from .__init__ import E, PY2, WINDOWS, ANYWIN, MACOS, VT100, unicode from .__init__ import PY2, WINDOWS, MACOS, VT100
from .util import mp, start_log_thrs, start_stackmon, min_ex from .util import mp
from .authsrv import AuthSrv from .authsrv import AuthSrv
from .tcpsrv import TcpSrv from .tcpsrv import TcpSrv
from .up2k import Up2k from .up2k import Up2k
@@ -32,31 +28,17 @@ class SvcHub(object):
put() can return a queue (if want_reply=True) which has a blocking get() with the response. put() can return a queue (if want_reply=True) which has a blocking get() with the response.
""" """
def __init__(self, args, argv, printed): def __init__(self, args):
self.args = args self.args = args
self.argv = argv
self.logf = None
self.stop_req = False
self.stopping = False
self.stop_cond = threading.Condition()
self.httpsrv_up = 0
self.ansi_re = re.compile("\033\\[[^m]*m") self.ansi_re = re.compile("\033\\[[^m]*m")
self.log_mutex = threading.Lock() self.log_mutex = threading.Lock()
self.next_day = 0 self.next_day = 0
self.log = self._log_disabled if args.q else self._log_enabled self.log = self._log_disabled if args.q else self._log_enabled
if args.lo:
self._setup_logfile(printed)
if args.stackmon:
start_stackmon(args.stackmon, 0)
if args.log_thrs:
start_log_thrs(self.log, args.log_thrs, 0)
# initiate all services to manage # initiate all services to manage
self.asrv = AuthSrv(self.args, self.log) self.asrv = AuthSrv(self.args, self.log, False)
if args.ls: if args.ls:
self.asrv.dbg_ls() self.asrv.dbg_ls()
@@ -87,138 +69,22 @@ class SvcHub(object):
self.broker = Broker(self) self.broker = Broker(self)
def thr_httpsrv_up(self):
time.sleep(5)
failed = self.broker.num_workers - self.httpsrv_up
if not failed:
return
m = "{}/{} workers failed to start"
m = m.format(failed, self.broker.num_workers)
self.log("root", m, 1)
os._exit(1)
def cb_httpsrv_up(self):
self.httpsrv_up += 1
if self.httpsrv_up != self.broker.num_workers:
return
self.log("root", "workers OK\n")
self.up2k.init_vols()
thr = threading.Thread(target=self.sd_notify, name="sd-notify")
thr.daemon = True
thr.start()
def _logname(self):
dt = datetime.utcfromtimestamp(time.time())
fn = self.args.lo
for fs in "YmdHMS":
fs = "%" + fs
if fs in fn:
fn = fn.replace(fs, dt.strftime(fs))
return fn
def _setup_logfile(self, printed):
base_fn = fn = sel_fn = self._logname()
if fn != self.args.lo:
ctr = 0
# yup this is a race; if started sufficiently concurrently, two
# copyparties can grab the same logfile (considered and ignored)
while os.path.exists(sel_fn):
ctr += 1
sel_fn = "{}.{}".format(fn, ctr)
fn = sel_fn
try:
import lzma
lh = lzma.open(fn, "wt", encoding="utf-8", errors="replace", preset=0)
except:
import codecs
lh = codecs.open(fn, "w", encoding="utf-8", errors="replace")
lh.base_fn = base_fn
argv = [sys.executable] + self.argv
if hasattr(shlex, "quote"):
argv = [shlex.quote(x) for x in argv]
else:
argv = ['"{}"'.format(x) for x in argv]
msg = "[+] opened logfile [{}]\n".format(fn)
printed += msg
lh.write("t0: {:.3f}\nargv: {}\n\n{}".format(E.t0, " ".join(argv), printed))
self.logf = lh
print(msg, end="")
def run(self): def run(self):
self.tcpsrv.run() thr = threading.Thread(target=self.tcpsrv.run, name="svchub-main")
thr = threading.Thread(target=self.thr_httpsrv_up)
thr.daemon = True thr.daemon = True
thr.start() thr.start()
for sig in [signal.SIGINT, signal.SIGTERM]: # winxp/py2.7 support: thr.join() kills signals
signal.signal(sig, self.signal_handler)
# macos hangs after shutdown on sigterm with while-sleep,
# windows cannot ^c stop_cond (and win10 does the macos thing but winxp is fine??)
# linux is fine with both,
# never lucky
if ANYWIN:
# msys-python probably fine but >msys-python
thr = threading.Thread(target=self.stop_thr, name="svchub-sig")
thr.daemon = True
thr.start()
try:
while not self.stop_req:
time.sleep(1)
except:
pass
self.shutdown()
thr.join()
else:
self.stop_thr()
def stop_thr(self):
while not self.stop_req:
with self.stop_cond:
self.stop_cond.wait(9001)
self.shutdown()
def signal_handler(self, sig, frame):
if self.stopping:
return
self.stop_req = True
with self.stop_cond:
self.stop_cond.notify_all()
def shutdown(self):
if self.stopping:
return
self.stopping = True
self.stop_req = True
with self.stop_cond:
self.stop_cond.notify_all()
ret = 1
try: try:
while True:
time.sleep(9001)
except KeyboardInterrupt:
with self.log_mutex: with self.log_mutex:
print("OPYTHAT") print("OPYTHAT")
self.tcpsrv.shutdown() self.tcpsrv.shutdown()
self.broker.shutdown() self.broker.shutdown()
self.up2k.shutdown()
if self.thumbsrv: if self.thumbsrv:
self.thumbsrv.shutdown() self.thumbsrv.shutdown()
@@ -231,41 +97,11 @@ class SvcHub(object):
print("waiting for thumbsrv (10sec)...") print("waiting for thumbsrv (10sec)...")
print("nailed it", end="") print("nailed it", end="")
ret = 0
finally: finally:
print("\033[0m") print("\033[0m")
if self.logf:
self.logf.close()
sys.exit(ret)
def _log_disabled(self, src, msg, c=0): def _log_disabled(self, src, msg, c=0):
if not self.logf: pass
return
with self.log_mutex:
ts = datetime.utcfromtimestamp(time.time())
ts = ts.strftime("%Y-%m%d-%H%M%S.%f")[:-3]
self.logf.write("@{} [{}] {}\n".format(ts, src, msg))
now = time.time()
if now >= self.next_day:
self._set_next_day()
def _set_next_day(self):
if self.next_day and self.logf and self.logf.base_fn != self._logname():
self.logf.close()
self._setup_logfile("")
dt = datetime.utcfromtimestamp(time.time())
# unix timestamp of next 00:00:00 (leap-seconds safe)
day_now = dt.day
while dt.day == day_now:
dt += timedelta(hours=12)
dt = dt.replace(hour=0, minute=0, second=0)
self.next_day = calendar.timegm(dt.utctimetuple())
def _log_enabled(self, src, msg, c=0): def _log_enabled(self, src, msg, c=0):
"""handles logging from all components""" """handles logging from all components"""
@@ -274,7 +110,14 @@ class SvcHub(object):
if now >= self.next_day: if now >= self.next_day:
dt = datetime.utcfromtimestamp(now) dt = datetime.utcfromtimestamp(now)
print("\033[36m{}\033[0m\n".format(dt.strftime("%Y-%m-%d")), end="") print("\033[36m{}\033[0m\n".format(dt.strftime("%Y-%m-%d")), end="")
self._set_next_day()
# unix timestamp of next 00:00:00 (leap-seconds safe)
day_now = dt.day
while dt.day == day_now:
dt += timedelta(hours=12)
dt = dt.replace(hour=0, minute=0, second=0)
self.next_day = calendar.timegm(dt.utctimetuple())
fmt = "\033[36m{} \033[33m{:21} \033[0m{}\n" fmt = "\033[36m{} \033[33m{:21} \033[0m{}\n"
if not VT100: if not VT100:
@@ -301,20 +144,20 @@ class SvcHub(object):
except: except:
print(msg.encode("ascii", "replace").decode(), end="") print(msg.encode("ascii", "replace").decode(), end="")
if self.logf:
self.logf.write(msg)
def check_mp_support(self): def check_mp_support(self):
vmin = sys.version_info[1] vmin = sys.version_info[1]
if WINDOWS: if WINDOWS:
msg = "need python 3.3 or newer for multiprocessing;" msg = "need python 3.3 or newer for multiprocessing;"
if PY2 or vmin < 3: if PY2:
# py2 pickler doesn't support winsock
return msg
elif vmin < 3:
return msg return msg
elif MACOS: elif MACOS:
return "multiprocessing is wonky on mac osx;" return "multiprocessing is wonky on mac osx;"
else: else:
msg = "need python 3.3+ for multiprocessing;" msg = "need python 2.7 or 3.3+ for multiprocessing;"
if PY2 or vmin < 3: if not PY2 and vmin < 3:
return msg return msg
try: try:
@@ -346,24 +189,5 @@ class SvcHub(object):
if not err: if not err:
return True return True
else: else:
self.log("svchub", err) self.log("root", err)
return False return False
def sd_notify(self):
try:
addr = os.getenv("NOTIFY_SOCKET")
if not addr:
return
addr = unicode(addr)
if addr.startswith("@"):
addr = "\0" + addr[1:]
m = "".join(x for x in addr if x in string.printable)
self.log("sd_notify", m)
sck = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
sck.connect(addr)
sck.sendall(b"READY=1")
except:
self.log("sd_notify", min_ex())

View File

@@ -4,15 +4,15 @@ from __future__ import print_function, unicode_literals
import os import os
import time import time
import zlib import zlib
import struct
from datetime import datetime from datetime import datetime
from .sutil import errdesc from .sutil import errdesc
from .util import yieldfile, sanitize_fn, spack, sunpack from .util import yieldfile, sanitize_fn
from .bos import bos
def dostime2unix(buf): def dostime2unix(buf):
t, d = sunpack(b"<HH", buf) t, d = struct.unpack("<HH", buf)
ts = (t & 0x1F) * 2 ts = (t & 0x1F) * 2
tm = (t >> 5) & 0x3F tm = (t >> 5) & 0x3F
@@ -36,13 +36,13 @@ def unixtime2dos(ts):
bd = ((dy - 1980) << 9) + (dm << 5) + dd bd = ((dy - 1980) << 9) + (dm << 5) + dd
bt = (th << 11) + (tm << 5) + ts // 2 bt = (th << 11) + (tm << 5) + ts // 2
return spack(b"<HH", bt, bd) return struct.pack("<HH", bt, bd)
def gen_fdesc(sz, crc32, z64): def gen_fdesc(sz, crc32, z64):
ret = b"\x50\x4b\x07\x08" ret = b"\x50\x4b\x07\x08"
fmt = b"<LQQ" if z64 else b"<LLL" fmt = "<LQQ" if z64 else "<LLL"
ret += spack(fmt, crc32, sz, sz) ret += struct.pack(fmt, crc32, sz, sz)
return ret return ret
@@ -66,7 +66,7 @@ def gen_hdr(h_pos, fn, sz, lastmod, utf8, crc32, pre_crc):
req_ver = b"\x2d\x00" if z64 else b"\x0a\x00" req_ver = b"\x2d\x00" if z64 else b"\x0a\x00"
if crc32: if crc32:
crc32 = spack(b"<L", crc32) crc32 = struct.pack("<L", crc32)
else: else:
crc32 = b"\x00" * 4 crc32 = b"\x00" * 4
@@ -87,14 +87,14 @@ def gen_hdr(h_pos, fn, sz, lastmod, utf8, crc32, pre_crc):
# however infozip does actual sz and it even works on winxp # however infozip does actual sz and it even works on winxp
# (same reasning for z64 extradata later) # (same reasning for z64 extradata later)
vsz = 0xFFFFFFFF if z64 else sz vsz = 0xFFFFFFFF if z64 else sz
ret += spack(b"<LL", vsz, vsz) ret += struct.pack("<LL", vsz, vsz)
# windows support (the "?" replace below too) # windows support (the "?" replace below too)
fn = sanitize_fn(fn, "/", []) fn = sanitize_fn(fn, ok="/")
bfn = fn.encode("utf-8" if utf8 else "cp437", "replace").replace(b"?", b"_") bfn = fn.encode("utf-8" if utf8 else "cp437", "replace").replace(b"?", b"_")
z64_len = len(z64v) * 8 + 4 if z64v else 0 z64_len = len(z64v) * 8 + 4 if z64v else 0
ret += spack(b"<HH", len(bfn), z64_len) ret += struct.pack("<HH", len(bfn), z64_len)
if h_pos is not None: if h_pos is not None:
# 2b comment, 2b diskno # 2b comment, 2b diskno
@@ -106,12 +106,12 @@ def gen_hdr(h_pos, fn, sz, lastmod, utf8, crc32, pre_crc):
ret += b"\x01\x00\x00\x00\xa4\x81" ret += b"\x01\x00\x00\x00\xa4\x81"
# 4b local-header-ofs # 4b local-header-ofs
ret += spack(b"<L", min(h_pos, 0xFFFFFFFF)) ret += struct.pack("<L", min(h_pos, 0xFFFFFFFF))
ret += bfn ret += bfn
if z64v: if z64v:
ret += spack(b"<HH" + b"Q" * len(z64v), 1, len(z64v) * 8, *z64v) ret += struct.pack("<HH" + "Q" * len(z64v), 1, len(z64v) * 8, *z64v)
return ret return ret
@@ -136,7 +136,7 @@ def gen_ecdr(items, cdir_pos, cdir_end):
need_64 = nitems == 0xFFFF or 0xFFFFFFFF in [csz, cpos] need_64 = nitems == 0xFFFF or 0xFFFFFFFF in [csz, cpos]
# 2b tnfiles, 2b dnfiles, 4b dir sz, 4b dir pos # 2b tnfiles, 2b dnfiles, 4b dir sz, 4b dir pos
ret += spack(b"<HHLL", nitems, nitems, csz, cpos) ret += struct.pack("<HHLL", nitems, nitems, csz, cpos)
# 2b comment length # 2b comment length
ret += b"\x00\x00" ret += b"\x00\x00"
@@ -163,7 +163,7 @@ def gen_ecdr64(items, cdir_pos, cdir_end):
# 8b tnfiles, 8b dnfiles, 8b dir sz, 8b dir pos # 8b tnfiles, 8b dnfiles, 8b dir sz, 8b dir pos
cdir_sz = cdir_end - cdir_pos cdir_sz = cdir_end - cdir_pos
ret += spack(b"<QQQQ", len(items), len(items), cdir_sz, cdir_pos) ret += struct.pack("<QQQQ", len(items), len(items), cdir_sz, cdir_pos)
return ret return ret
@@ -178,14 +178,13 @@ def gen_ecdr64_loc(ecdr64_pos):
ret = b"\x50\x4b\x06\x07" ret = b"\x50\x4b\x06\x07"
# 4b cdisk, 8b start of ecdr64, 4b ndisks # 4b cdisk, 8b start of ecdr64, 4b ndisks
ret += spack(b"<LQL", 0, ecdr64_pos, 1) ret += struct.pack("<LQL", 0, ecdr64_pos, 1)
return ret return ret
class StreamZip(object): class StreamZip(object):
def __init__(self, log, fgen, utf8=False, pre_crc=False): def __init__(self, fgen, utf8=False, pre_crc=False):
self.log = log
self.fgen = fgen self.fgen = fgen
self.utf8 = utf8 self.utf8 = utf8
self.pre_crc = pre_crc self.pre_crc = pre_crc
@@ -248,8 +247,8 @@ class StreamZip(object):
errors.append([f["vp"], repr(ex)]) errors.append([f["vp"], repr(ex)])
if errors: if errors:
errf, txt = errdesc(errors) errf = errdesc(errors)
self.log("\n".join(([repr(errf)] + txt[1:]))) print(repr(errf))
for x in self.ser(errf): for x in self.ser(errf):
yield x yield x
@@ -272,4 +271,4 @@ class StreamZip(object):
yield self._ct(ecdr) yield self._ct(ecdr)
if errors: if errors:
bos.unlink(errf["ap"]) os.unlink(errf["ap"])

View File

@@ -2,10 +2,11 @@
from __future__ import print_function, unicode_literals from __future__ import print_function, unicode_literals
import re import re
import time
import socket import socket
import select
from .__init__ import MACOS, ANYWIN from .util import chkcmd, Counter
from .util import chkcmd
class TcpSrv(object): class TcpSrv(object):
@@ -19,6 +20,7 @@ class TcpSrv(object):
self.args = hub.args self.args = hub.args
self.log = hub.log self.log = hub.log
self.num_clients = Counter()
self.stopping = False self.stopping = False
ip = "127.0.0.1" ip = "127.0.0.1"
@@ -30,16 +32,14 @@ class TcpSrv(object):
for x in nonlocals: for x in nonlocals:
eps[x] = "external" eps[x] = "external"
msgs = []
m = "available @ http://{}:{}/ (\033[33m{}\033[0m)"
for ip, desc in sorted(eps.items(), key=lambda x: x[1]): for ip, desc in sorted(eps.items(), key=lambda x: x[1]):
for port in sorted(self.args.p): for port in sorted(self.args.p):
msgs.append(m.format(ip, port, desc)) self.log(
"tcpsrv",
if msgs: "available @ http://{}:{}/ (\033[33m{}\033[0m)".format(
msgs[-1] += "\n" ip, port, desc
for m in msgs: ),
self.log("tcpsrv", m) )
self.srv = [] self.srv = []
for ip in self.args.i: for ip in self.args.i:
@@ -66,13 +66,44 @@ class TcpSrv(object):
for srv in self.srv: for srv in self.srv:
srv.listen(self.args.nc) srv.listen(self.args.nc)
ip, port = srv.getsockname() ip, port = srv.getsockname()
fno = srv.fileno() self.log("tcpsrv", "listening @ {0}:{1}".format(ip, port))
msg = "listening @ {}:{} f{}".format(ip, port, fno)
self.log("tcpsrv", msg)
if self.args.q:
print(msg)
self.hub.broker.put(False, "listen", srv) while not self.stopping:
if self.args.log_conn:
self.log("tcpsrv", "|%sC-ncli" % ("-" * 1,), c="1;30")
if self.num_clients.v >= self.args.nc:
time.sleep(0.1)
continue
if self.args.log_conn:
self.log("tcpsrv", "|%sC-acc1" % ("-" * 2,), c="1;30")
try:
# macos throws bad-fd
ready, _, _ = select.select(self.srv, [], [])
except:
ready = []
if not self.stopping:
raise
for srv in ready:
if self.stopping:
break
sck, addr = srv.accept()
sip, sport = srv.getsockname()
if self.args.log_conn:
self.log(
"%s %s" % addr,
"|{}C-acc2 \033[0;36m{} \033[3{}m{}".format(
"-" * 3, sip, sport % 8, sport
),
c="1;30",
)
self.num_clients.add()
self.hub.broker.put(False, "httpconn", sck, addr)
def shutdown(self): def shutdown(self):
self.stopping = True self.stopping = True
@@ -84,100 +115,25 @@ class TcpSrv(object):
self.log("tcpsrv", "ok bye") self.log("tcpsrv", "ok bye")
def ips_linux(self):
eps = {}
try:
txt, _ = chkcmd(["ip", "addr"])
except:
return eps
r = re.compile(r"^\s+inet ([^ ]+)/.* (.*)")
for ln in txt.split("\n"):
try:
ip, dev = r.match(ln.rstrip()).groups()
eps[ip] = dev
except:
pass
return eps
def ips_macos(self):
eps = {}
try:
txt, _ = chkcmd(["ifconfig"])
except:
return eps
rdev = re.compile(r"^([^ ]+):")
rip = re.compile(r"^\tinet ([0-9\.]+) ")
dev = None
for ln in txt.split("\n"):
m = rdev.match(ln)
if m:
dev = m.group(1)
m = rip.match(ln)
if m:
eps[m.group(1)] = dev
dev = None
return eps
def ips_windows_ipconfig(self):
eps = {}
try:
txt, _ = chkcmd(["ipconfig"])
except:
return eps
rdev = re.compile(r"(^[^ ].*):$")
rip = re.compile(r"^ +IPv?4? [^:]+: *([0-9\.]{7,15})$")
dev = None
for ln in txt.replace("\r", "").split("\n"):
m = rdev.match(ln)
if m:
dev = m.group(1).split(" adapter ", 1)[-1]
m = rip.match(ln)
if m and dev:
eps[m.group(1)] = dev
dev = None
return eps
def ips_windows_netsh(self):
eps = {}
try:
txt, _ = chkcmd("netsh interface ip show address".split())
except:
return eps
rdev = re.compile(r'.* "([^"]+)"$')
rip = re.compile(r".* IP\b.*: +([0-9\.]{7,15})$")
dev = None
for ln in txt.replace("\r", "").split("\n"):
m = rdev.match(ln)
if m:
dev = m.group(1)
m = rip.match(ln)
if m and dev:
eps[m.group(1)] = dev
dev = None
return eps
def detect_interfaces(self, listen_ips): def detect_interfaces(self, listen_ips):
if MACOS: eps = {}
eps = self.ips_macos()
elif ANYWIN:
eps = self.ips_windows_ipconfig() # sees more interfaces
eps.update(self.ips_windows_netsh()) # has better names
else:
eps = self.ips_linux()
if "0.0.0.0" not in listen_ips: # get all ips and their interfaces
eps = {k: v for k, v in eps if k in listen_ips} try:
ip_addr, _ = chkcmd("ip", "addr")
except:
ip_addr = None
if ip_addr:
r = re.compile(r"^\s+inet ([^ ]+)/.* (.*)")
for ln in ip_addr.split("\n"):
try:
ip, dev = r.match(ln.rstrip()).groups()
for lip in listen_ips:
if lip in ["0.0.0.0", ip]:
eps[ip] = dev
except:
pass
default_route = None default_route = None
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

View File

@@ -5,7 +5,6 @@ import os
from .util import Cooldown from .util import Cooldown
from .th_srv import thumb_path, THUMBABLE, FMT_FF from .th_srv import thumb_path, THUMBABLE, FMT_FF
from .bos import bos
class ThumbCli(object): class ThumbCli(object):
@@ -37,7 +36,7 @@ class ThumbCli(object):
tpath = thumb_path(histpath, rem, mtime, fmt) tpath = thumb_path(histpath, rem, mtime, fmt)
ret = None ret = None
try: try:
st = bos.stat(tpath) st = os.stat(tpath)
if st.st_size: if st.st_size:
ret = tpath ret = tpath
else: else:

View File

@@ -9,12 +9,15 @@ import hashlib
import threading import threading
import subprocess as sp import subprocess as sp
from .__init__ import PY2, unicode from .__init__ import PY2
from .util import fsenc, vsplit, runcmd, Queue, Cooldown, BytesIO, min_ex from .util import fsenc, runcmd, Queue, Cooldown, BytesIO, min_ex
from .bos import bos
from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, ffprobe from .mtag import HAVE_FFMPEG, HAVE_FFPROBE, ffprobe
if not PY2:
unicode = str
HAVE_PIL = False HAVE_PIL = False
HAVE_HEIF = False HAVE_HEIF = False
HAVE_AVIF = False HAVE_AVIF = False
@@ -50,7 +53,7 @@ except:
# https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html # https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html
# ffmpeg -formats # ffmpeg -formats
FMT_PIL = "bmp dib gif icns ico jpg jpeg jp2 jpx pcx png pbm pgm ppm pnm sgi tga tif tiff webp xbm dds xpm" FMT_PIL = "bmp dib gif icns ico jpg jpeg jp2 jpx pcx png pbm pgm ppm pnm sgi tga tif tiff webp xbm dds xpm"
FMT_FF = "av1 asf avi flv m4v mkv mjpeg mjpg mpg mpeg mpg2 mpeg2 h264 avc mts h265 hevc mov 3gp mp4 ts mpegts nut ogv ogm rm vob webm wmv" FMT_FF = "av1 asf avi flv m4v mkv mjpeg mjpg mpg mpeg mpg2 mpeg2 h264 avc h265 hevc mov 3gp mp4 ts mpegts nut ogv ogm rm vob webm wmv"
if HAVE_HEIF: if HAVE_HEIF:
FMT_PIL += " heif heifs heic heics" FMT_PIL += " heif heifs heic heics"
@@ -74,7 +77,12 @@ def thumb_path(histpath, rem, mtime, fmt):
# base16 = 16 = 256 # base16 = 16 = 256
# b64-lc = 38 = 1444 # b64-lc = 38 = 1444
# base64 = 64 = 4096 # base64 = 64 = 4096
rd, fn = vsplit(rem) try:
rd, fn = rem.rsplit("/", 1)
except:
rd = ""
fn = rem
if rd: if rd:
h = hashlib.sha512(fsenc(rd)).digest() h = hashlib.sha512(fsenc(rd)).digest()
b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24] b64 = base64.urlsafe_b64encode(h).decode("ascii")[:24]
@@ -117,19 +125,18 @@ class ThumbSrv(object):
if not self.args.no_vthumb and (not HAVE_FFMPEG or not HAVE_FFPROBE): if not self.args.no_vthumb and (not HAVE_FFMPEG or not HAVE_FFPROBE):
missing = [] missing = []
if not HAVE_FFMPEG: if not HAVE_FFMPEG:
missing.append("FFmpeg") missing.append("ffmpeg")
if not HAVE_FFPROBE: if not HAVE_FFPROBE:
missing.append("FFprobe") missing.append("ffprobe")
msg = "cannot create video thumbnails because some of the required programs are not available: " msg = "cannot create video thumbnails because some of the required programs are not available: "
msg += ", ".join(missing) msg += ", ".join(missing)
self.log(msg, c=3) self.log(msg, c=3)
if self.args.th_clean: t = threading.Thread(target=self.cleaner, name="thumb-cleaner")
t = threading.Thread(target=self.cleaner, name="thumb-cleaner") t.daemon = True
t.daemon = True t.start()
t.start()
def log(self, msg, c=0): def log(self, msg, c=0):
self.log_func("thumb", msg, c) self.log_func("thumb", msg, c)
@@ -155,10 +162,13 @@ class ThumbSrv(object):
self.log("wait {}".format(tpath)) self.log("wait {}".format(tpath))
except: except:
thdir = os.path.dirname(tpath) thdir = os.path.dirname(tpath)
bos.makedirs(thdir) try:
os.makedirs(thdir)
except:
pass
inf_path = os.path.join(thdir, "dir.txt") inf_path = os.path.join(thdir, "dir.txt")
if not bos.path.exists(inf_path): if not os.path.exists(inf_path):
with open(inf_path, "wb") as f: with open(inf_path, "wb") as f:
f.write(fsenc(os.path.dirname(abspath))) f.write(fsenc(os.path.dirname(abspath)))
@@ -178,7 +188,7 @@ class ThumbSrv(object):
cond.wait(3) cond.wait(3)
try: try:
st = bos.stat(tpath) st = os.stat(tpath)
if st.st_size: if st.st_size:
return tpath return tpath
except: except:
@@ -195,7 +205,7 @@ class ThumbSrv(object):
abspath, tpath = task abspath, tpath = task
ext = abspath.split(".")[-1].lower() ext = abspath.split(".")[-1].lower()
fun = None fun = None
if not bos.path.exists(tpath): if not os.path.exists(tpath):
if ext in FMT_PIL: if ext in FMT_PIL:
fun = self.conv_pil fun = self.conv_pil
elif ext in FMT_FF: elif ext in FMT_FF:
@@ -253,7 +263,7 @@ class ThumbSrv(object):
pass # default q = 75 pass # default q = 75
if im.mode not in fmts: if im.mode not in fmts:
# print("conv {}".format(im.mode)) print("conv {}".format(im.mode))
im = im.convert("RGB") im = im.convert("RGB")
im.save(tpath, quality=40, method=6) im.save(tpath, quality=40, method=6)
@@ -306,7 +316,7 @@ class ThumbSrv(object):
cmd += [fsenc(tpath)] cmd += [fsenc(tpath)]
ret, sout, serr = runcmd(cmd) ret, sout, serr = runcmd(*cmd)
if ret != 0: if ret != 0:
msg = ["ff: {}".format(x) for x in serr.split("\n")] msg = ["ff: {}".format(x) for x in serr.split("\n")]
self.log("FFmpeg failed:\n" + "\n".join(msg), c="1;30") self.log("FFmpeg failed:\n" + "\n".join(msg), c="1;30")
@@ -321,7 +331,7 @@ class ThumbSrv(object):
p1 = os.path.dirname(tdir) p1 = os.path.dirname(tdir)
p2 = os.path.dirname(p1) p2 = os.path.dirname(p1)
for dp in [tdir, p1, p2]: for dp in [tdir, p1, p2]:
bos.utime(dp, (ts, ts)) os.utime(fsenc(dp), (ts, ts))
except: except:
pass pass
@@ -348,7 +358,7 @@ class ThumbSrv(object):
prev_b64 = None prev_b64 = None
prev_fp = None prev_fp = None
try: try:
ents = bos.listdir(thumbpath) ents = os.listdir(thumbpath)
except: except:
return 0 return 0
@@ -359,7 +369,7 @@ class ThumbSrv(object):
# "top" or b64 prefix/full (a folder) # "top" or b64 prefix/full (a folder)
if len(f) <= 3 or len(f) == 24: if len(f) <= 3 or len(f) == 24:
age = now - bos.path.getmtime(fp) age = now - os.path.getmtime(fp)
if age > maxage: if age > maxage:
with self.mutex: with self.mutex:
safe = True safe = True
@@ -391,7 +401,7 @@ class ThumbSrv(object):
if b64 == prev_b64: if b64 == prev_b64:
self.log("rm replaced [{}]".format(fp)) self.log("rm replaced [{}]".format(fp))
bos.unlink(prev_fp) os.unlink(prev_fp)
prev_b64 = b64 prev_b64 = b64
prev_fp = fp prev_fp = fp

View File

@@ -7,9 +7,7 @@ import time
import threading import threading
from datetime import datetime from datetime import datetime
from .__init__ import unicode
from .util import s3dec, Pebkac, min_ex from .util import s3dec, Pebkac, min_ex
from .bos import bos
from .up2k import up2k_wark_from_hashlist from .up2k import up2k_wark_from_hashlist
@@ -68,7 +66,7 @@ class U2idx(object):
histpath = self.asrv.vfs.histtab[ptop] histpath = self.asrv.vfs.histtab[ptop]
db_path = os.path.join(histpath, "up2k.db") db_path = os.path.join(histpath, "up2k.db")
if not bos.path.exists(db_path): if not os.path.exists(db_path):
return None return None
cur = sqlite3.connect(db_path, 2).cursor() cur = sqlite3.connect(db_path, 2).cursor()
@@ -92,8 +90,6 @@ class U2idx(object):
mt_ctr = 0 mt_ctr = 0
mt_keycmp = "substr(up.w,1,16)" mt_keycmp = "substr(up.w,1,16)"
mt_keycmp2 = None mt_keycmp2 = None
ptn_lc = re.compile(r" (mt[0-9]+\.v) ([=<!>]+) \? $")
ptn_lcv = re.compile(r"[a-zA-Z]")
while True: while True:
uq = uq.strip() uq = uq.strip()
@@ -186,21 +182,6 @@ class U2idx(object):
va.append(v) va.append(v)
is_key = True is_key = True
# lowercase tag searches
m = ptn_lc.search(q)
if not m or not ptn_lcv.search(unicode(v)):
continue
va.pop()
va.append(v.lower())
q = q[: m.start()]
field, oper = m.groups()
if oper in ["=", "=="]:
q += " {} like ? ".format(field)
else:
q += " lower({}) {} ? ".format(field, oper)
try: try:
return self.run_query(vols, joins + "where " + q, va) return self.run_query(vols, joins + "where " + q, va)
except Exception as ex: except Exception as ex:

View File

@@ -23,20 +23,15 @@ from .util import (
ProgressPrinter, ProgressPrinter,
fsdec, fsdec,
fsenc, fsenc,
absreal,
sanitize_fn, sanitize_fn,
ren_open, ren_open,
atomic_move, atomic_move,
vsplit,
s3enc, s3enc,
s3dec, s3dec,
rmdirs,
statdir, statdir,
s2hms, s2hms,
min_ex, min_ex,
) )
from .bos import bos
from .authsrv import AuthSrv
from .mtag import MTag, MParser from .mtag import MTag, MParser
try: try:
@@ -49,9 +44,16 @@ DB_VER = 4
class Up2k(object): class Up2k(object):
"""
TODO:
* documentation
* registry persistence
* ~/.config flatfiles for active jobs
"""
def __init__(self, hub): def __init__(self, hub):
self.hub = hub self.hub = hub
self.asrv = hub.asrv # type: AuthSrv self.asrv = hub.asrv
self.args = hub.args self.args = hub.args
self.log_func = hub.log self.log_func = hub.log
@@ -65,7 +67,6 @@ class Up2k(object):
self.n_hashq = 0 self.n_hashq = 0
self.n_tagq = 0 self.n_tagq = 0
self.volstate = {} self.volstate = {}
self.need_rescan = {}
self.registry = {} self.registry = {}
self.entags = {} self.entags = {}
self.flags = {} self.flags = {}
@@ -100,14 +101,13 @@ class Up2k(object):
if self.args.no_fastboot: if self.args.no_fastboot:
self.deferred_init() self.deferred_init()
else:
def init_vols(self): t = threading.Thread(
if self.args.no_fastboot: target=self.deferred_init,
return name="up2k-deferred-init",
)
t = threading.Thread(target=self.deferred_init, name="up2k-deferred-init") t.daemon = True
t.daemon = True t.start()
t.start()
def deferred_init(self): def deferred_init(self):
all_vols = self.asrv.vfs.all_vols all_vols = self.asrv.vfs.all_vols
@@ -122,10 +122,6 @@ class Up2k(object):
thr.daemon = True thr.daemon = True
thr.start() thr.start()
thr = threading.Thread(target=self._sched_rescan, name="up2k-rescan")
thr.daemon = True
thr.start()
if self.mtag: if self.mtag:
thr = threading.Thread(target=self._tagger, name="up2k-tagger") thr = threading.Thread(target=self._tagger, name="up2k-tagger")
thr.daemon = True thr.daemon = True
@@ -175,38 +171,6 @@ class Up2k(object):
t.start() t.start()
return None return None
def _sched_rescan(self):
maxage = self.args.re_maxage
volage = {}
while True:
time.sleep(self.args.re_int)
now = time.time()
vpaths = list(sorted(self.asrv.vfs.all_vols.keys()))
with self.mutex:
if maxage:
for vp in vpaths:
if vp not in volage:
volage[vp] = now
if now - volage[vp] >= maxage:
self.need_rescan[vp] = 1
if not self.need_rescan:
continue
vols = list(sorted(self.need_rescan.keys()))
self.need_rescan = {}
err = self.rescan(self.asrv.vfs.all_vols, vols)
if err:
for v in vols:
self.need_rescan[v] = True
continue
for v in vols:
volage[v] = now
def _vis_job_progress(self, job): def _vis_job_progress(self, job):
perc = 100 - (len(job["need"]) * 100.0 / len(job["hash"])) perc = 100 - (len(job["need"]) * 100.0 / len(job["hash"]))
path = os.path.join(job["ptop"], job["prel"], job["name"]) path = os.path.join(job["ptop"], job["prel"], job["name"])
@@ -229,7 +193,7 @@ class Up2k(object):
return True, ret return True, ret
def init_indexes(self, all_vols, scan_vols=None): def init_indexes(self, all_vols, scan_vols=[]):
self.pp = ProgressPrinter() self.pp = ProgressPrinter()
vols = all_vols.values() vols = all_vols.values()
t0 = time.time() t0 = time.time()
@@ -252,7 +216,7 @@ class Up2k(object):
# only need to protect register_vpath but all in one go feels right # only need to protect register_vpath but all in one go feels right
for vol in vols: for vol in vols:
try: try:
bos.listdir(vol.realpath) os.listdir(vol.realpath)
except: except:
self.volstate[vol.vpath] = "OFFLINE (cannot access folder)" self.volstate[vol.vpath] = "OFFLINE (cannot access folder)"
self.log("cannot access " + vol.realpath, c=1) self.log("cannot access " + vol.realpath, c=1)
@@ -338,7 +302,7 @@ class Up2k(object):
self.log(msg.format(len(vols), time.time() - t0)) self.log(msg.format(len(vols), time.time() - t0))
if needed_mutagen: if needed_mutagen:
msg = "could not read tags because no backends are available (Mutagen or FFprobe)" msg = "could not read tags because no backends are available (mutagen or ffprobe)"
self.log(msg, c=1) self.log(msg, c=1)
thr = None thr = None
@@ -378,26 +342,18 @@ class Up2k(object):
for k, v in flags.items() for k, v in flags.items()
] ]
if a: if a:
vpath = "?" self.log(" ".join(sorted(a)) + "\033[0m")
for k, v in self.asrv.vfs.all_vols.items():
if v.realpath == ptop:
vpath = k
if vpath:
vpath += "/"
self.log("/{} {}".format(vpath, " ".join(sorted(a))), "35")
reg = {} reg = {}
path = os.path.join(histpath, "up2k.snap") path = os.path.join(histpath, "up2k.snap")
if "e2d" in flags and bos.path.exists(path): if "e2d" in flags and os.path.exists(path):
with gzip.GzipFile(path, "rb") as f: with gzip.GzipFile(path, "rb") as f:
j = f.read().decode("utf-8") j = f.read().decode("utf-8")
reg2 = json.loads(j) reg2 = json.loads(j)
for k, job in reg2.items(): for k, job in reg2.items():
path = os.path.join(job["ptop"], job["prel"], job["name"]) path = os.path.join(job["ptop"], job["prel"], job["name"])
if bos.path.exists(path): if os.path.exists(fsenc(path)):
reg[k] = job reg[k] = job
job["poke"] = time.time() job["poke"] = time.time()
else: else:
@@ -412,7 +368,10 @@ class Up2k(object):
if not HAVE_SQLITE3 or "e2d" not in flags or "d2d" in flags: if not HAVE_SQLITE3 or "e2d" not in flags or "d2d" in flags:
return None return None
bos.makedirs(histpath) try:
os.makedirs(histpath)
except:
pass
try: try:
cur = self._open_db(db_path) cur = self._open_db(db_path)
@@ -442,7 +401,7 @@ class Up2k(object):
if WINDOWS: if WINDOWS:
excl = [x.replace("/", "\\") for x in excl] excl = [x.replace("/", "\\") for x in excl]
n_add = self._build_dir(dbw, top, set(excl), top, nohash, []) n_add = self._build_dir(dbw, top, set(excl), top, nohash)
n_rm = self._drop_lost(dbw[0], top) n_rm = self._drop_lost(dbw[0], top)
if dbw[1]: if dbw[1]:
self.log("commit {} new files".format(dbw[1])) self.log("commit {} new files".format(dbw[1]))
@@ -450,18 +409,11 @@ class Up2k(object):
return True, n_add or n_rm or do_vac return True, n_add or n_rm or do_vac
def _build_dir(self, dbw, top, excl, cdir, nohash, seen): def _build_dir(self, dbw, top, excl, cdir, nohash):
rcdir = absreal(cdir) # a bit expensive but worth
if rcdir in seen:
m = "bailing from symlink loop,\n prev: {}\n curr: {}\n from: {}"
self.log(m.format(seen[-1], rcdir, cdir), 3)
return 0
seen = seen + [cdir]
self.pp.msg = "a{} {}".format(self.pp.n, cdir) self.pp.msg = "a{} {}".format(self.pp.n, cdir)
histpath = self.asrv.vfs.histtab[top] histpath = self.asrv.vfs.histtab[top]
ret = 0 ret = 0
g = statdir(self.log_func, not self.args.no_scandir, False, cdir) g = statdir(self.log, not self.args.no_scandir, False, cdir)
for iname, inf in sorted(g): for iname, inf in sorted(g):
abspath = os.path.join(cdir, iname) abspath = os.path.join(cdir, iname)
lmod = int(inf.st_mtime) lmod = int(inf.st_mtime)
@@ -470,7 +422,7 @@ class Up2k(object):
if abspath in excl or abspath == histpath: if abspath in excl or abspath == histpath:
continue continue
# self.log(" dir: {}".format(abspath)) # self.log(" dir: {}".format(abspath))
ret += self._build_dir(dbw, top, excl, abspath, nohash, seen) ret += self._build_dir(dbw, top, excl, abspath, nohash)
else: else:
# self.log("file: {}".format(abspath)) # self.log("file: {}".format(abspath))
rp = abspath[len(top) + 1 :] rp = abspath[len(top) + 1 :]
@@ -547,7 +499,7 @@ class Up2k(object):
# almost zero overhead dw # almost zero overhead dw
self.pp.msg = "b{} {}".format(nfiles - nchecked, abspath) self.pp.msg = "b{} {}".format(nfiles - nchecked, abspath)
try: try:
if not bos.path.exists(abspath): if not os.path.exists(fsenc(abspath)):
rm.append([drd, dfn]) rm.append([drd, dfn])
except Exception as ex: except Exception as ex:
self.log("stat-rm: {} @ [{}]".format(repr(ex), abspath)) self.log("stat-rm: {} @ [{}]".format(repr(ex), abspath))
@@ -620,7 +572,7 @@ class Up2k(object):
c2 = conn.cursor() c2 = conn.cursor()
c3 = conn.cursor() c3 = conn.cursor()
n_left = cur.execute("select count(w) from up").fetchone()[0] n_left = cur.execute("select count(w) from up").fetchone()[0]
for w, rd, fn in cur.execute("select w, rd, fn from up order by rd, fn"): for w, rd, fn in cur.execute("select w, rd, fn from up"):
n_left -= 1 n_left -= 1
q = "select w from mt where w = ?" q = "select w from mt where w = ?"
if c2.execute(q, (w[:16],)).fetchone(): if c2.execute(q, (w[:16],)).fetchone():
@@ -935,7 +887,7 @@ class Up2k(object):
# x.set_trace_callback(trace) # x.set_trace_callback(trace)
def _open_db(self, db_path): def _open_db(self, db_path):
existed = bos.path.exists(db_path) existed = os.path.exists(db_path)
cur = self._orz(db_path) cur = self._orz(db_path)
ver = self._read_ver(cur) ver = self._read_ver(cur)
if not existed and ver is None: if not existed and ver is None:
@@ -953,38 +905,19 @@ class Up2k(object):
m = "database is version {}, this copyparty only supports versions <= {}" m = "database is version {}, this copyparty only supports versions <= {}"
raise Exception(m.format(ver, DB_VER)) raise Exception(m.format(ver, DB_VER))
bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
db = cur.connection
cur.close()
db.close()
msg = "creating new DB (old is bad); backup: {}" msg = "creating new DB (old is bad); backup: {}"
if ver: if ver:
msg = "creating new DB (too old to upgrade); backup: {}" msg = "creating new DB (too old to upgrade); backup: {}"
cur = self._backup_db(db_path, cur, ver, msg) self.log(msg.format(bak))
db = cur.connection os.rename(fsenc(db_path), fsenc(bak))
cur.close()
db.close()
bos.unlink(db_path)
return self._create_db(db_path, None) return self._create_db(db_path, None)
def _backup_db(self, db_path, cur, ver, msg):
bak = "{}.bak.{:x}.v{}".format(db_path, int(time.time()), ver)
self.log(msg + bak)
try:
c2 = sqlite3.connect(bak)
with c2:
cur.connection.backup(c2)
return cur
except:
m = "native sqlite3 backup failed; using fallback method:\n"
self.log(m + min_ex())
finally:
c2.close()
db = cur.connection
cur.close()
db.close()
shutil.copy2(fsenc(db_path), fsenc(bak))
return self._orz(db_path)
def _read_ver(self, cur): def _read_ver(self, cur):
for tab in ["ki", "kv"]: for tab in ["ki", "kv"]:
try: try:
@@ -1034,7 +967,7 @@ class Up2k(object):
if cj["ptop"] not in self.registry: if cj["ptop"] not in self.registry:
raise Pebkac(410, "location unavailable") raise Pebkac(410, "location unavailable")
cj["name"] = sanitize_fn(cj["name"], "", [".prologue.html", ".epilogue.html"]) cj["name"] = sanitize_fn(cj["name"], bad=[".prologue.html", ".epilogue.html"])
cj["poke"] = time.time() cj["poke"] = time.time()
wark = self._get_wark(cj) wark = self._get_wark(cj)
now = time.time() now = time.time()
@@ -1057,7 +990,7 @@ class Up2k(object):
dp_abs = "/".join([cj["ptop"], dp_dir, dp_fn]) dp_abs = "/".join([cj["ptop"], dp_dir, dp_fn])
# relying on path.exists to return false on broken symlinks # relying on path.exists to return false on broken symlinks
if bos.path.exists(dp_abs): if os.path.exists(fsenc(dp_abs)):
job = { job = {
"name": dp_fn, "name": dp_fn,
"prel": dp_dir, "prel": dp_dir,
@@ -1081,7 +1014,7 @@ class Up2k(object):
for fn in names: for fn in names:
path = os.path.join(job["ptop"], job["prel"], fn) path = os.path.join(job["ptop"], job["prel"], fn)
try: try:
if bos.path.getsize(path) > 0: if os.path.getsize(fsenc(path)) > 0:
# upload completed or both present # upload completed or both present
break break
except: except:
@@ -1114,14 +1047,8 @@ class Up2k(object):
pdir = os.path.join(cj["ptop"], cj["prel"]) pdir = os.path.join(cj["ptop"], cj["prel"])
job["name"] = self._untaken(pdir, cj["name"], now, cj["addr"]) job["name"] = self._untaken(pdir, cj["name"], now, cj["addr"])
dst = os.path.join(job["ptop"], job["prel"], job["name"]) dst = os.path.join(job["ptop"], job["prel"], job["name"])
if not self.args.nw: os.unlink(fsenc(dst)) # TODO ed pls
bos.unlink(dst) # TODO ed pls self._symlink(src, dst)
self._symlink(src, dst)
if cur:
a = [cj[x] for x in "prel name lmod size".split()]
self.db_add(cur, wark, *a)
cur.connection.commit()
if not job: if not job:
job = { job = {
@@ -1172,18 +1099,17 @@ class Up2k(object):
with ren_open(fname, "wb", fdir=fdir, suffix=suffix) as f: with ren_open(fname, "wb", fdir=fdir, suffix=suffix) as f:
return f["orz"][1] return f["orz"][1]
def _symlink(self, src, dst, verbose=True): def _symlink(self, src, dst):
if verbose: # TODO store this in linktab so we never delete src if there are links to it
self.log("linking dupe:\n {0}\n {1}".format(src, dst)) self.log("linking dupe:\n {0}\n {1}".format(src, dst))
if self.args.nw: if self.args.nw:
return return
try: try:
lsrc = src lsrc = src
ldst = dst ldst = dst
fs1 = bos.stat(os.path.dirname(src)).st_dev fs1 = os.stat(fsenc(os.path.split(src)[0])).st_dev
fs2 = bos.stat(os.path.dirname(dst)).st_dev fs2 = os.stat(fsenc(os.path.split(dst)[0])).st_dev
if fs1 == 0: if fs1 == 0:
# py2 on winxp or other unsupported combination # py2 on winxp or other unsupported combination
raise OSError() raise OSError()
@@ -1266,8 +1192,15 @@ class Up2k(object):
a = [dst, job["size"], (int(time.time()), int(job["lmod"]))] a = [dst, job["size"], (int(time.time()), int(job["lmod"]))]
self.lastmod_q.put(a) self.lastmod_q.put(a)
a = [job[x] for x in "ptop wark prel name lmod size".split()] # legit api sware 2 me mum
if self.idx_wark(*a): if self.idx_wark(
job["ptop"],
job["wark"],
job["prel"],
job["name"],
job["lmod"],
job["size"],
):
del self.registry[ptop][wark] del self.registry[ptop][wark]
# in-memory registry is reserved for unfinished uploads # in-memory registry is reserved for unfinished uploads
@@ -1279,7 +1212,7 @@ class Up2k(object):
return False return False
self.db_rm(cur, rd, fn) self.db_rm(cur, rd, fn)
self.db_add(cur, wark, rd, fn, lmod, sz) self.db_add(cur, wark, rd, fn, int(lmod), sz)
cur.connection.commit() cur.connection.commit()
if "e2t" in self.flags[ptop]: if "e2t" in self.flags[ptop]:
@@ -1302,251 +1235,9 @@ class Up2k(object):
db.execute(sql, v) db.execute(sql, v)
except: except:
rd, fn = s3enc(self.mem_cur, rd, fn) rd, fn = s3enc(self.mem_cur, rd, fn)
v = (wark, int(ts), sz, rd, fn) v = (wark, ts, sz, rd, fn)
db.execute(sql, v) db.execute(sql, v)
def handle_rm(self, uname, vpath):
permsets = [[True, False, False, True]]
vn, rem = self.asrv.vfs.get(vpath, uname, *permsets[0])
ptop = vn.realpath
atop = vn.canonical(rem)
adir, fn = os.path.split(atop)
st = bos.lstat(atop)
scandir = not self.args.no_scandir
if stat.S_ISLNK(st.st_mode) or stat.S_ISREG(st.st_mode):
dbv, vrem = self.asrv.vfs.get(vpath, uname, *permsets[0])
dbv, vrem = dbv.get_dbv(vrem)
g = [[dbv, vrem, os.path.dirname(vpath), adir, [[fn, 0]], [], []]]
else:
g = vn.walk("", rem, [], uname, permsets, True, scandir, True)
n_files = 0
for dbv, vrem, _, adir, files, rd, vd in g:
for fn in [x[0] for x in files]:
n_files += 1
abspath = os.path.join(adir, fn)
vpath = "{}/{}".format(vrem, fn).strip("/")
self.log("rm {}\n {}".format(vpath, abspath))
_ = dbv.get(vrem, uname, *permsets[0])
with self.mutex:
try:
ptop = dbv.realpath
cur, wark, _, _ = self._find_from_vpath(ptop, vrem)
self._forget_file(ptop, vpath, cur, wark)
finally:
cur.connection.commit()
bos.unlink(abspath)
rm = rmdirs(self.log_func, scandir, True, atop)
ok = len(rm[0])
ng = len(rm[1])
return "deleted {} files (and {}/{} folders)".format(n_files, ok, ok + ng)
def handle_mv(self, uname, svp, dvp):
svn, srem = self.asrv.vfs.get(svp, uname, True, False, True)
svn, srem = svn.get_dbv(srem)
sabs = svn.canonical(srem, False)
if not srem:
raise Pebkac(400, "mv: cannot move a mountpoint")
st = bos.stat(sabs)
if stat.S_ISREG(st.st_mode):
return self._mv_file(uname, svp, dvp)
jail = svn.get_dbv(srem)[0]
permsets = [[True, False, True]]
scandir = not self.args.no_scandir
# following symlinks is too scary
g = svn.walk("", srem, [], uname, permsets, True, scandir, True)
for dbv, vrem, _, atop, files, rd, vd in g:
if dbv != jail:
# fail early (prevent partial moves)
raise Pebkac(400, "mv: source folder contains other volumes")
g = svn.walk("", srem, [], uname, permsets, True, scandir, True)
for dbv, vrem, _, atop, files, rd, vd in g:
if dbv != jail:
# the actual check (avoid toctou)
raise Pebkac(400, "mv: source folder contains other volumes")
for fn in files:
svpf = "/".join(x for x in [dbv.vpath, vrem, fn[0]] if x)
if not svpf.startswith(svp + "/"): # assert
raise Pebkac(500, "mv: bug at {}, top {}".format(svpf, svp))
dvpf = dvp + svpf[len(svp) :]
self._mv_file(uname, svpf, dvpf)
rmdirs(self.log_func, scandir, True, sabs)
return "k"
def _mv_file(self, uname, svp, dvp):
svn, srem = self.asrv.vfs.get(svp, uname, True, False, True)
svn, srem = svn.get_dbv(srem)
dvn, drem = self.asrv.vfs.get(dvp, uname, False, True)
dvn, drem = dvn.get_dbv(drem)
sabs = svn.canonical(srem, False)
dabs = dvn.canonical(drem)
drd, dfn = vsplit(drem)
if bos.path.exists(dabs):
raise Pebkac(400, "mv2: target file exists")
bos.makedirs(os.path.dirname(dabs))
if bos.path.islink(sabs):
dlabs = absreal(sabs)
m = "moving symlink from [{}] to [{}], target [{}]"
self.log(m.format(sabs, dabs, dlabs))
os.unlink(sabs)
self._symlink(dlabs, dabs, False)
# folders are too scary, schedule rescan of both vols
self.need_rescan[svn.vpath] = 1
self.need_rescan[dvn.vpath] = 1
return "k"
c1, w, ftime, fsize = self._find_from_vpath(svn.realpath, srem)
c2 = self.cur.get(dvn.realpath)
if ftime is None:
st = bos.stat(sabs)
ftime = st.st_mtime
fsize = st.st_size
if w:
if c2:
self._copy_tags(c1, c2, w)
self._forget_file(svn.realpath, srem, c1, w)
self._relink(w, svn.realpath, srem, dabs)
c1.connection.commit()
if c2:
self.db_add(c2, w, drd, dfn, ftime, fsize)
c2.connection.commit()
else:
self.log("not found in src db: [{}]".format(svp))
bos.rename(sabs, dabs)
return "k"
def _copy_tags(self, csrc, cdst, wark):
"""copy all tags for wark from src-db to dst-db"""
w = wark[:16]
if cdst.execute("select * from mt where w=? limit 1", (w,)).fetchone():
return # existing tags in dest db
for _, k, v in csrc.execute("select * from mt where w=?", (w,)):
cdst.execute("insert into mt values(?,?,?)", (w, k, v))
def _find_from_vpath(self, ptop, vrem):
cur = self.cur.get(ptop)
if not cur:
return None, None
rd, fn = vsplit(vrem)
q = "select w, mt, sz from up where rd=? and fn=? limit 1"
try:
c = cur.execute(q, (rd, fn))
except:
c = cur.execute(q, s3enc(self.mem_cur, rd, fn))
hit = c.fetchone()
if hit:
wark, ftime, fsize = hit
return cur, wark, ftime, fsize
return cur, None, None, None
def _forget_file(self, ptop, vrem, cur, wark):
"""forgets file in db, fixes symlinks, does not delete"""
srd, sfn = vsplit(vrem)
self.log("forgetting {}".format(vrem))
if wark:
self.log("found {} in db".format(wark))
self._relink(wark, ptop, vrem, None)
q = "delete from mt where w=?"
cur.execute(q, (wark[:16],))
self.db_rm(cur, srd, sfn)
reg = self.registry.get(ptop)
if reg:
if not wark:
wark = [
x
for x, y in reg.items()
if fn in [y["name"], y.get("tnam")] and y["prel"] == vrem
]
if wark and wark in reg:
m = "forgetting partial upload {} ({})"
p = self._vis_job_progress(wark)
self.log(m.format(wark, p))
del reg[wark]
def _relink(self, wark, sptop, srem, dabs):
"""
update symlinks from file at svn/srem to dabs (rename),
or to first remaining full if no dabs (delete)
"""
dupes = []
sabs = os.path.join(sptop, srem)
q = "select rd, fn from up where substr(w,1,16)=? and w=?"
for ptop, cur in self.cur.items():
for rd, fn in cur.execute(q, (wark[:16], wark)):
if rd.startswith("//") or fn.startswith("//"):
rd, fn = s3dec(rd, fn)
dvrem = "/".join([rd, fn]).strip("/")
if ptop != sptop or srem != dvrem:
dupes.append([ptop, dvrem])
self.log("found {} dupe: [{}] {}".format(wark, ptop, dvrem))
if not dupes:
return
full = {}
links = {}
for ptop, vp in dupes:
ap = os.path.join(ptop, vp)
try:
d = links if bos.path.islink(ap) else full
d[ap] = [ptop, vp]
except:
self.log("relink: not found: [{}]".format(ap))
if not dabs and not full and links:
# deleting final remaining full copy; swap it with a symlink
slabs = list(sorted(links.keys()))[0]
ptop, rem = links.pop(slabs)
self.log("linkswap [{}] and [{}]".format(sabs, dabs))
bos.unlink(slabs)
bos.rename(sabs, slabs)
self._symlink(slabs, sabs, False)
full[slabs] = [ptop, rem]
if not dabs:
dabs = list(sorted(full.keys()))[0]
for alink in links.keys():
try:
if alink != sabs and absreal(alink) != sabs:
continue
self.log("relinking [{}] to [{}]".format(alink, dabs))
bos.unlink(alink)
except:
pass
self._symlink(dabs, alink, False)
def _get_wark(self, cj): def _get_wark(self, cj):
if len(cj["name"]) > 1024 or len(cj["hash"]) > 512 * 1024: # 16TiB if len(cj["name"]) > 1024 or len(cj["hash"]) > 512 * 1024: # 16TiB
raise Pebkac(400, "name or numchunks not according to spec") raise Pebkac(400, "name or numchunks not according to spec")
@@ -1568,7 +1259,7 @@ class Up2k(object):
def _hashlist_from_file(self, path): def _hashlist_from_file(self, path):
pp = self.pp if hasattr(self, "pp") else None pp = self.pp if hasattr(self, "pp") else None
fsz = bos.path.getsize(path) fsz = os.path.getsize(fsenc(path))
csz = up2k_chunksize(fsz) csz = up2k_chunksize(fsz)
ret = [] ret = []
with open(fsenc(path), "rb", 512 * 1024) as f: with open(fsenc(path), "rb", 512 * 1024) as f:
@@ -1636,7 +1327,7 @@ class Up2k(object):
for path, sz, times in ready: for path, sz, times in ready:
self.log("lmod: setting times {} on {}".format(times, path)) self.log("lmod: setting times {} on {}".format(times, path))
try: try:
bos.utime(path, times) os.utime(fsenc(path), times)
except: except:
self.log("lmod: failed to utime ({}, {})".format(path, times)) self.log("lmod: failed to utime ({}, {})".format(path, times))
@@ -1647,22 +1338,19 @@ class Up2k(object):
self.log("could not unsparse [{}]".format(path), 3) self.log("could not unsparse [{}]".format(path), 3)
def _snapshot(self): def _snapshot(self):
self.snap_persist_interval = 300 # persist unfinished index every 5 min persist_interval = 30 # persist unfinished uploads index every 30 sec
self.snap_discard_interval = 21600 # drop unfinished after 6 hours inactivity discard_interval = 21600 # drop unfinished uploads after 6 hours inactivity
self.snap_prev = {} prev = {}
while True: while True:
time.sleep(self.snap_persist_interval) time.sleep(persist_interval)
self.do_snapshot() with self.mutex:
for k, reg in self.registry.items():
self._snap_reg(prev, k, reg, discard_interval)
def do_snapshot(self): def _snap_reg(self, prev, ptop, reg, discard_interval):
with self.mutex:
for k, reg in self.registry.items():
self._snap_reg(k, reg)
def _snap_reg(self, ptop, reg):
now = time.time() now = time.time()
histpath = self.asrv.vfs.histtab[ptop] histpath = self.asrv.vfs.histtab[ptop]
rm = [x for x in reg.values() if now - x["poke"] > self.snap_discard_interval] rm = [x for x in reg.values() if now - x["poke"] > discard_interval]
if rm: if rm:
m = "dropping {} abandoned uploads in {}".format(len(rm), ptop) m = "dropping {} abandoned uploads in {}".format(len(rm), ptop)
vis = [self._vis_job_progress(x) for x in rm] vis = [self._vis_job_progress(x) for x in rm]
@@ -1672,30 +1360,33 @@ class Up2k(object):
try: try:
# remove the filename reservation # remove the filename reservation
path = os.path.join(job["ptop"], job["prel"], job["name"]) path = os.path.join(job["ptop"], job["prel"], job["name"])
if bos.path.getsize(path) == 0: if os.path.getsize(fsenc(path)) == 0:
bos.unlink(path) os.unlink(fsenc(path))
if len(job["hash"]) == len(job["need"]): if len(job["hash"]) == len(job["need"]):
# PARTIAL is empty, delete that too # PARTIAL is empty, delete that too
path = os.path.join(job["ptop"], job["prel"], job["tnam"]) path = os.path.join(job["ptop"], job["prel"], job["tnam"])
bos.unlink(path) os.unlink(fsenc(path))
except: except:
pass pass
path = os.path.join(histpath, "up2k.snap") path = os.path.join(histpath, "up2k.snap")
if not reg: if not reg:
if ptop not in self.snap_prev or self.snap_prev[ptop] is not None: if ptop not in prev or prev[ptop] is not None:
self.snap_prev[ptop] = None prev[ptop] = None
if bos.path.exists(path): if os.path.exists(fsenc(path)):
bos.unlink(path) os.unlink(fsenc(path))
return return
newest = max(x["poke"] for _, x in reg.items()) if reg else 0 newest = max(x["poke"] for _, x in reg.items()) if reg else 0
etag = [len(reg), newest] etag = [len(reg), newest]
if etag == self.snap_prev.get(ptop): if etag == prev.get(ptop):
return return
bos.makedirs(histpath) try:
os.makedirs(histpath)
except:
pass
path2 = "{}.{}".format(path, os.getpid()) path2 = "{}.{}".format(path, os.getpid())
j = json.dumps(reg, indent=2, sort_keys=True).encode("utf-8") j = json.dumps(reg, indent=2, sort_keys=True).encode("utf-8")
@@ -1705,7 +1396,7 @@ class Up2k(object):
atomic_move(path2, path) atomic_move(path2, path)
self.log("snap: {} |{}|".format(path, len(reg.keys()))) self.log("snap: {} |{}|".format(path, len(reg.keys())))
self.snap_prev[ptop] = etag prev[ptop] = etag
def _tagger(self): def _tagger(self):
with self.mutex: with self.mutex:
@@ -1760,7 +1451,7 @@ class Up2k(object):
abspath = os.path.join(ptop, rd, fn) abspath = os.path.join(ptop, rd, fn)
self.log("hashing " + abspath) self.log("hashing " + abspath)
inf = bos.stat(abspath) inf = os.stat(fsenc(abspath))
hashes = self._hashlist_from_file(abspath) hashes = self._hashlist_from_file(abspath)
wark = up2k_wark_from_hashlist(self.salt, inf.st_size, hashes) wark = up2k_wark_from_hashlist(self.salt, inf.st_size, hashes)
with self.mutex: with self.mutex:
@@ -1773,11 +1464,6 @@ class Up2k(object):
self.n_hashq += 1 self.n_hashq += 1
# self.log("hashq {} push {}/{}/{}".format(self.n_hashq, ptop, rd, fn)) # self.log("hashq {} push {}/{}/{}".format(self.n_hashq, ptop, rd, fn))
def shutdown(self):
if hasattr(self, "snap_prev"):
self.log("writing snapshot")
self.do_snapshot()
def up2k_chunksize(filesize): def up2k_chunksize(filesize):
chunksize = 1024 * 1024 chunksize = 1024 * 1024
@@ -1793,7 +1479,7 @@ def up2k_chunksize(filesize):
def up2k_wark_from_hashlist(salt, filesize, hashes): def up2k_wark_from_hashlist(salt, filesize, hashes):
"""server-reproducible file identifier, independent of name or location""" """ server-reproducible file identifier, independent of name or location """
ident = [salt, str(filesize)] ident = [salt, str(filesize)]
ident.extend(hashes) ident.extend(hashes)
ident = "\n".join(ident) ident = "\n".join(ident)

View File

@@ -4,7 +4,6 @@ from __future__ import print_function, unicode_literals
import re import re
import os import os
import sys import sys
import stat
import time import time
import base64 import base64
import select import select
@@ -17,7 +16,6 @@ import mimetypes
import contextlib import contextlib
import subprocess as sp # nosec import subprocess as sp # nosec
from datetime import datetime from datetime import datetime
from collections import Counter
from .__init__ import PY2, WINDOWS, ANYWIN from .__init__ import PY2, WINDOWS, ANYWIN
from .stolen import surrogateescape from .stolen import surrogateescape
@@ -44,20 +42,6 @@ else:
from Queue import Queue # pylint: disable=import-error,no-name-in-module from Queue import Queue # pylint: disable=import-error,no-name-in-module
from StringIO import StringIO as BytesIO from StringIO import StringIO as BytesIO
try:
struct.unpack(b">i", b"idgi")
spack = struct.pack
sunpack = struct.unpack
except:
def spack(f, *a, **ka):
return struct.pack(f.decode("ascii"), *a, **ka)
def sunpack(f, *a, **ka):
return struct.unpack(f.decode("ascii"), *a, **ka)
surrogateescape.register_surrogateescape() surrogateescape.register_surrogateescape()
FS_ENCODING = sys.getfilesystemencoding() FS_ENCODING = sys.getfilesystemencoding()
if WINDOWS and PY2: if WINDOWS and PY2:
@@ -139,6 +123,20 @@ REKOBO_KEY = {
REKOBO_LKEY = {k.lower(): v for k, v in REKOBO_KEY.items()} REKOBO_LKEY = {k.lower(): v for k, v in REKOBO_KEY.items()}
class Counter(object):
def __init__(self, v=0):
self.v = v
self.mutex = threading.Lock()
def add(self, delta=1):
with self.mutex:
self.v += delta
def set(self, absval):
with self.mutex:
self.v = absval
class Cooldown(object): class Cooldown(object):
def __init__(self, maxage): def __init__(self, maxage):
self.maxage = maxage self.maxage = maxage
@@ -233,7 +231,7 @@ def nuprint(msg):
def rice_tid(): def rice_tid():
tid = threading.current_thread().ident tid = threading.current_thread().ident
c = sunpack(b"B" * 5, spack(b">Q", tid)[-5:]) c = struct.unpack(b"B" * 5, struct.pack(b">Q", tid)[-5:])
return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c) + "\033[0m" return "".join("\033[1;37;48;5;{}m{:02x}".format(x, x) for x in c) + "\033[0m"
@@ -284,69 +282,15 @@ def alltrace():
return "\n".join(rret + bret) return "\n".join(rret + bret)
def start_stackmon(arg_str, nid):
suffix = "-{}".format(nid) if nid else ""
fp, f = arg_str.rsplit(",", 1)
f = int(f)
t = threading.Thread(
target=stackmon,
args=(fp, f, suffix),
name="stackmon" + suffix,
)
t.daemon = True
t.start()
def stackmon(fp, ival, suffix):
ctr = 0
while True:
ctr += 1
time.sleep(ival)
st = "{}, {}\n{}".format(ctr, time.time(), alltrace())
with open(fp + suffix, "wb") as f:
f.write(st.encode("utf-8", "replace"))
def start_log_thrs(logger, ival, nid):
ival = int(ival)
tname = lname = "log-thrs"
if nid:
tname = "logthr-n{}-i{:x}".format(nid, os.getpid())
lname = tname[3:]
t = threading.Thread(
target=log_thrs,
args=(logger, ival, lname),
name=tname,
)
t.daemon = True
t.start()
def log_thrs(log, ival, name):
while True:
time.sleep(ival)
tv = [x.name for x in threading.enumerate()]
tv = [
x.split("-")[0]
if x.startswith("httpconn-") or x.startswith("thumb-")
else "listen"
if "-listen-" in x
else x
for x in tv
if not x.startswith("pydevd.")
]
tv = ["{}\033[36m{}".format(v, k) for k, v in sorted(Counter(tv).items())]
log(name, "\033[0m \033[33m".join(tv), 3)
def min_ex(): def min_ex():
et, ev, tb = sys.exc_info() et, ev, tb = sys.exc_info()
tb = traceback.extract_tb(tb) tb = traceback.extract_tb(tb, 2)
fmt = "{} @ {} <{}>: {}" ex = [
ex = [fmt.format(fp.split(os.sep)[-1], ln, fun, txt) for fp, ln, fun, txt in tb] "{} @ {} <{}>: {}".format(fp.split(os.sep)[-1], ln, fun, txt)
ex.append("[{}] {}".format(et.__name__, ev)) for fp, ln, fun, txt in tb
return "\n".join(ex[-8:]) ]
ex.append("{}: {}".format(et.__name__, ev))
return "\n".join(ex)
@contextlib.contextmanager @contextlib.contextmanager
@@ -730,7 +674,7 @@ def undot(path):
return "/".join(ret) return "/".join(ret)
def sanitize_fn(fn, ok, bad): def sanitize_fn(fn, ok="", bad=[]):
if "/" not in ok: if "/" not in ok:
fn = fn.replace("\\", "/").split("/")[-1] fn = fn.replace("\\", "/").split("/")[-1]
@@ -759,19 +703,6 @@ def sanitize_fn(fn, ok, bad):
return fn.strip() return fn.strip()
def absreal(fpath):
try:
return fsdec(os.path.abspath(os.path.realpath(fsenc(fpath))))
except:
if not WINDOWS:
raise
# cpython bug introduced in 3.8, still exists in 3.9.1,
# some win7sp1 and win10:20H2 boxes cannot realpath a
# networked drive letter such as b"n:" or b"n:\\"
return os.path.abspath(os.path.realpath(fpath))
def u8safe(txt): def u8safe(txt):
try: try:
return txt.encode("utf-8", "xmlcharrefreplace").decode("utf-8", "replace") return txt.encode("utf-8", "xmlcharrefreplace").decode("utf-8", "replace")
@@ -829,13 +760,6 @@ def unquotep(txt):
return w8dec(unq2) return w8dec(unq2)
def vsplit(vpath):
if "/" not in vpath:
return "", vpath
return vpath.rsplit("/", 1)
def w8dec(txt): def w8dec(txt):
"""decodes filesystem-bytes to wtf8""" """decodes filesystem-bytes to wtf8"""
if PY2: if PY2:
@@ -980,10 +904,16 @@ def yieldfile(fn):
yield buf yield buf
def hashcopy(fin, fout): def hashcopy(actor, fin, fout):
is_mp = actor.is_mp
hashobj = hashlib.sha512() hashobj = hashlib.sha512()
tlen = 0 tlen = 0
for buf in fin: for buf in fin:
if is_mp:
actor.workload += 1
if actor.workload > 2 ** 31:
actor.workload = 100
tlen += len(buf) tlen += len(buf)
hashobj.update(buf) hashobj.update(buf)
fout.write(buf) fout.write(buf)
@@ -994,10 +924,15 @@ def hashcopy(fin, fout):
return tlen, hashobj.hexdigest(), digest_b64 return tlen, hashobj.hexdigest(), digest_b64
def sendfile_py(lower, upper, f, s): def sendfile_py(lower, upper, f, s, actor=None):
remains = upper - lower remains = upper - lower
f.seek(lower) f.seek(lower)
while remains > 0: while remains > 0:
if actor:
actor.workload += 1
if actor.workload > 2 ** 31:
actor.workload = 100
# time.sleep(0.01) # time.sleep(0.01)
buf = f.read(min(1024 * 32, remains)) buf = f.read(min(1024 * 32, remains))
if not buf: if not buf:
@@ -1035,9 +970,6 @@ def sendfile_kern(lower, upper, f, s):
def statdir(logger, scandir, lstat, top): def statdir(logger, scandir, lstat, top):
if lstat and not os.supports_follow_symlinks:
scandir = False
try: try:
btop = fsenc(top) btop = fsenc(top)
if scandir and hasattr(os, "scandir"): if scandir and hasattr(os, "scandir"):
@@ -1047,7 +979,8 @@ def statdir(logger, scandir, lstat, top):
try: try:
yield [fsdec(fh.name), fh.stat(follow_symlinks=not lstat)] yield [fsdec(fh.name), fh.stat(follow_symlinks=not lstat)]
except Exception as ex: except Exception as ex:
logger(src, "[s] {} @ {}".format(repr(ex), fsdec(fh.path)), 6) msg = "scan-stat: \033[36m{} @ {}"
logger(msg.format(repr(ex), fsdec(fh.path)))
else: else:
src = "listdir" src = "listdir"
fun = os.lstat if lstat else os.stat fun = os.lstat if lstat else os.stat
@@ -1056,30 +989,11 @@ def statdir(logger, scandir, lstat, top):
try: try:
yield [fsdec(name), fun(abspath)] yield [fsdec(name), fun(abspath)]
except Exception as ex: except Exception as ex:
logger(src, "[s] {} @ {}".format(repr(ex), fsdec(abspath)), 6) msg = "list-stat: \033[36m{} @ {}"
logger(msg.format(repr(ex), fsdec(abspath)))
except Exception as ex: except Exception as ex:
logger(src, "{} @ {}".format(repr(ex), top), 1) logger("{}: \033[31m{} @ {}".format(src, repr(ex), top))
def rmdirs(logger, scandir, lstat, top):
dirs = statdir(logger, scandir, lstat, top)
dirs = [x[0] for x in dirs if stat.S_ISDIR(x[1].st_mode)]
dirs = [os.path.join(top, x) for x in dirs]
ok = []
ng = []
for d in dirs[::-1]:
a, b = rmdirs(logger, scandir, lstat, d)
ok += a
ng += b
try:
os.rmdir(fsenc(top))
ok.append(top)
except:
ng.append(top)
return ok, ng
def unescape_cookie(orig): def unescape_cookie(orig):
@@ -1121,11 +1035,11 @@ def guess_mime(url, fallback="application/octet-stream"):
if ";" not in ret: if ";" not in ret:
if ret.startswith("text/") or ret.endswith("/javascript"): if ret.startswith("text/") or ret.endswith("/javascript"):
ret += "; charset=UTF-8" ret += "; charset=UTF-8"
return ret return ret
def runcmd(argv): def runcmd(*argv):
p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE) p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = p.communicate() stdout, stderr = p.communicate()
stdout = stdout.decode("utf-8", "replace") stdout = stdout.decode("utf-8", "replace")
@@ -1133,8 +1047,8 @@ def runcmd(argv):
return [p.returncode, stdout, stderr] return [p.returncode, stdout, stderr]
def chkcmd(argv): def chkcmd(*argv):
ok, sout, serr = runcmd(argv) ok, sout, serr = runcmd(*argv)
if ok != 0: if ok != 0:
raise Exception(serr) raise Exception(serr)
@@ -1156,7 +1070,10 @@ def gzip_orig_sz(fn):
with open(fsenc(fn), "rb") as f: with open(fsenc(fn), "rb") as f:
f.seek(-4, 2) f.seek(-4, 2)
rv = f.read(4) rv = f.read(4)
return sunpack(b"I", rv)[0] try:
return struct.unpack(b"I", rv)[0]
except:
return struct.unpack("I", rv)[0]
def py_desc(): def py_desc():

80
copyparty/vcr.py Normal file
View File

@@ -0,0 +1,80 @@
# coding: utf-8
from __future__ import print_function, unicode_literals
import time
import shlex
import subprocess as sp
from .__init__ import PY2
from .util import fsenc
class VCR_Direct(object):
def __init__(self, cli, fpath):
self.cli = cli
self.fpath = fpath
self.log_func = cli.log_func
self.log_src = cli.log_src
def log(self, msg, c=0):
self.log_func(self.log_src, "vcr: {}".format(msg), c)
def run(self):
opts = self.cli.uparam
# fmt: off
cmd = [
"ffmpeg",
"-nostdin",
"-hide_banner",
"-v", "warning",
"-i", fsenc(self.fpath),
"-vf", "scale=640:-4",
"-c:a", "libopus",
"-b:a", "128k",
"-c:v", "libvpx",
"-deadline", "realtime",
"-row-mt", "1"
]
# fmt: on
if "ss" in opts:
cmd.extend(["-ss", opts["ss"]])
if "crf" in opts:
cmd.extend(["-b:v", "0", "-crf", opts["crf"]])
else:
cmd.extend(["-b:v", "{}M".format(opts.get("mbps", 1.2))])
cmd.extend(["-f", "webm", "-"])
comp = str if not PY2 else unicode
cmd = [x.encode("utf-8") if isinstance(x, comp) else x for x in cmd]
self.log(" ".join([shlex.quote(x.decode("utf-8", "replace")) for x in cmd]))
p = sp.Popen(cmd, stdout=sp.PIPE)
self.cli.send_headers(None, mime="video/webm")
fails = 0
while True:
self.log("read")
buf = p.stdout.read(1024 * 4)
if not buf:
fails += 1
if p.poll() is not None or fails > 30:
self.log("ffmpeg exited")
return
time.sleep(0.1)
continue
fails = 0
try:
self.cli.s.sendall(buf)
except:
self.log("client disconnected")
p.kill()
return

View File

@@ -13,7 +13,7 @@ window.baguetteBox = (function () {
captions: true, captions: true,
buttons: 'auto', buttons: 'auto',
noScrollbars: false, noScrollbars: false,
bodyClass: 'bbox-open', bodyClass: 'baguetteBox-open',
titleTag: false, titleTag: false,
async: false, async: false,
preload: 2, preload: 2,
@@ -22,46 +22,37 @@ window.baguetteBox = (function () {
afterHide: null, afterHide: null,
onChange: null, onChange: null,
}, },
overlay, slider, btnPrev, btnNext, btnHelp, btnVmode, btnClose, overlay, slider, previousButton, nextButton, closeButton,
currentGallery = [], currentGallery = [],
currentIndex = 0, currentIndex = 0,
isOverlayVisible = false, isOverlayVisible = false,
touch = {}, // start-pos touch = {}, // start-pos
touchFlag = false, // busy touchFlag = false, // busy
re_i = /.+\.(gif|jpe?g|png|webp)(\?|$)/i, regex = /.+\.(gif|jpe?g|png|webp)/i,
re_v = /.+\.(webm|mp4)(\?|$)/i,
data = {}, // all galleries data = {}, // all galleries
imagesElements = [], imagesElements = [],
documentLastFocus = null, documentLastFocus = null;
isFullscreen = false,
vmute = false,
vloop = false,
vnext = false,
resume_mp = false;
var onFSC = function (e) { var overlayClickHandler = function (event) {
isFullscreen = !!document.fullscreenElement; if (event.target.id.indexOf('baguette-img') !== -1) {
};
var overlayClickHandler = function (e) {
if (e.target.id.indexOf('baguette-img') !== -1)
hideOverlay(); hideOverlay();
}
}; };
var touchstartHandler = function (e) { var touchstartHandler = function (event) {
touch.count++; touch.count++;
if (touch.count > 1) if (touch.count > 1) {
touch.multitouch = true; touch.multitouch = true;
}
touch.startX = e.changedTouches[0].pageX; touch.startX = event.changedTouches[0].pageX;
touch.startY = e.changedTouches[0].pageY; touch.startY = event.changedTouches[0].pageY;
}; };
var touchmoveHandler = function (e) { var touchmoveHandler = function (event) {
if (touchFlag || touch.multitouch) if (touchFlag || touch.multitouch) {
return; return;
}
e.preventDefault ? e.preventDefault() : e.returnValue = false; event.preventDefault ? event.preventDefault() : event.returnValue = false;
var touchEvent = e.touches[0] || e.changedTouches[0]; var touchEvent = event.touches[0] || event.changedTouches[0];
if (touchEvent.pageX - touch.startX > 40) { if (touchEvent.pageX - touch.startX > 40) {
touchFlag = true; touchFlag = true;
showPreviousImage(); showPreviousImage();
@@ -74,19 +65,19 @@ window.baguetteBox = (function () {
}; };
var touchendHandler = function () { var touchendHandler = function () {
touch.count--; touch.count--;
if (touch.count <= 0) if (touch.count <= 0) {
touch.multitouch = false; touch.multitouch = false;
}
touchFlag = false; touchFlag = false;
}; };
var contextmenuHandler = function () { var contextmenuHandler = function () {
touchendHandler(); touchendHandler();
}; };
var trapFocusInsideOverlay = function (e) { var trapFocusInsideOverlay = function (event) {
if (overlay.style.display === 'block' && (overlay.contains && !overlay.contains(e.target))) { if (overlay.style.display === 'block' && (overlay.contains && !overlay.contains(event.target))) {
e.stopPropagation(); event.stopPropagation();
btnClose.focus(); initFocus();
} }
}; };
@@ -97,7 +88,7 @@ window.baguetteBox = (function () {
} }
function bindImageClickListeners(selector, userOptions) { function bindImageClickListeners(selector, userOptions) {
var galleryNodeList = QSA(selector); var galleryNodeList = document.querySelectorAll(selector);
var selectorData = { var selectorData = {
galleries: [], galleries: [],
nodeList: galleryNodeList nodeList: galleryNodeList
@@ -105,26 +96,33 @@ window.baguetteBox = (function () {
data[selector] = selectorData; data[selector] = selectorData;
[].forEach.call(galleryNodeList, function (galleryElement) { [].forEach.call(galleryNodeList, function (galleryElement) {
if (userOptions && userOptions.filter) {
regex = userOptions.filter;
}
var tagsNodeList = []; var tagsNodeList = [];
if (galleryElement.tagName === 'A') if (galleryElement.tagName === 'A') {
tagsNodeList = [galleryElement]; tagsNodeList = [galleryElement];
else } else {
tagsNodeList = galleryElement.getElementsByTagName('a'); tagsNodeList = galleryElement.getElementsByTagName('a');
}
tagsNodeList = [].filter.call(tagsNodeList, function (element) { tagsNodeList = [].filter.call(tagsNodeList, function (element) {
if (element.className.indexOf(userOptions && userOptions.ignoreClass) === -1) if (element.className.indexOf(userOptions && userOptions.ignoreClass) === -1) {
return re_i.test(element.href) || re_v.test(element.href); return regex.test(element.href);
}
}); });
if (!tagsNodeList.length) if (tagsNodeList.length === 0) {
return; return;
}
var gallery = []; var gallery = [];
[].forEach.call(tagsNodeList, function (imageElement, imageIndex) { [].forEach.call(tagsNodeList, function (imageElement, imageIndex) {
var imageElementClickHandler = function (e) { var imageElementClickHandler = function (event) {
if (ctrl(e)) if (event && (event.ctrlKey || event.metaKey))
return true; return true;
e.preventDefault ? e.preventDefault() : e.returnValue = false; event.preventDefault ? event.preventDefault() : event.returnValue = false;
prepareOverlay(gallery, userOptions); prepareOverlay(gallery, userOptions);
showOverlay(imageIndex); showOverlay(imageIndex);
}; };
@@ -142,186 +140,93 @@ window.baguetteBox = (function () {
} }
function clearCachedData() { function clearCachedData() {
for (var selector in data) for (var selector in data) {
if (data.hasOwnProperty(selector)) if (data.hasOwnProperty(selector)) {
removeFromCache(selector); removeFromCache(selector);
}
}
} }
function removeFromCache(selector) { function removeFromCache(selector) {
if (!data.hasOwnProperty(selector)) if (!data.hasOwnProperty(selector)) {
return; return;
}
var galleries = data[selector].galleries; var galleries = data[selector].galleries;
[].forEach.call(galleries, function (gallery) { [].forEach.call(galleries, function (gallery) {
[].forEach.call(gallery, function (imageItem) { [].forEach.call(gallery, function (imageItem) {
unbind(imageItem.imageElement, 'click', imageItem.eventHandler); unbind(imageItem.imageElement, 'click', imageItem.eventHandler);
}); });
if (currentGallery === gallery) if (currentGallery === gallery) {
currentGallery = []; currentGallery = [];
}
}); });
delete data[selector]; delete data[selector];
} }
function buildOverlay() { function buildOverlay() {
overlay = ebi('bbox-overlay'); overlay = ebi('baguetteBox-overlay');
if (!overlay) { if (overlay) {
var ctr = mknod('div'); slider = ebi('baguetteBox-slider');
ctr.innerHTML = ( previousButton = ebi('previous-button');
'<div id="bbox-overlay" role="dialog">' + nextButton = ebi('next-button');
'<div id="bbox-slider"></div>' + closeButton = ebi('close-button');
'<button id="bbox-prev" class="bbox-btn" type="button" aria-label="Previous">&lt;</button>' + return;
'<button id="bbox-next" class="bbox-btn" type="button" aria-label="Next">&gt;</button>' +
'<div id="bbox-btns">' +
'<button id="bbox-help" type="button">?</button>' +
'<button id="bbox-vmode" type="button" tt="a"></button>' +
'<button id="bbox-close" type="button" aria-label="Close">X</button>' +
'</div></div>'
);
overlay = ctr.firstChild;
QS('body').appendChild(overlay);
tt.att(overlay);
} }
slider = ebi('bbox-slider'); overlay = mknod('div');
btnPrev = ebi('bbox-prev'); overlay.setAttribute('role', 'dialog');
btnNext = ebi('bbox-next'); overlay.id = 'baguetteBox-overlay';
btnHelp = ebi('bbox-help'); document.getElementsByTagName('body')[0].appendChild(overlay);
btnVmode = ebi('bbox-vmode');
btnClose = ebi('bbox-close'); slider = mknod('div');
slider.id = 'baguetteBox-slider';
overlay.appendChild(slider);
previousButton = mknod('button');
previousButton.setAttribute('type', 'button');
previousButton.id = 'previous-button';
previousButton.setAttribute('aria-label', 'Previous');
previousButton.innerHTML = '&lt;';
overlay.appendChild(previousButton);
nextButton = mknod('button');
nextButton.setAttribute('type', 'button');
nextButton.id = 'next-button';
nextButton.setAttribute('aria-label', 'Next');
nextButton.innerHTML = '&gt;';
overlay.appendChild(nextButton);
closeButton = mknod('button');
closeButton.setAttribute('type', 'button');
closeButton.id = 'close-button';
closeButton.setAttribute('aria-label', 'Close');
closeButton.innerHTML = '&times;';
overlay.appendChild(closeButton);
previousButton.className = nextButton.className = closeButton.className = 'baguetteBox-button';
bindEvents(); bindEvents();
} }
function halp() { function keyDownHandler(event) {
if (ebi('bbox-halp')) switch (event.keyCode) {
return; case 37: // Left
showPreviousImage();
var list = [ break;
['<b># hotkey</b>', '<b># operation</b>'], case 39: // Right
['escape', 'close'], showNextImage();
['left, J', 'previous file'], break;
['right, L', 'next file'], case 27: // Esc
['home', 'first file'], hideOverlay();
['end', 'last file'], break;
['space, P, K', 'video: play / pause'], case 36: // Home
['U', 'video: seek 10sec back'], showFirstImage(event);
['P', 'video: seek 10sec ahead'], break;
['M', 'video: toggle mute'], case 35: // End
['R', 'video: toggle loop'], showLastImage(event);
['C', 'video: toggle auto-next'], break;
['F', 'video: toggle fullscreen'],
],
d = mknod('table'),
html = ['<tbody>'];
for (var a = 0; a < list.length; a++)
html.push('<tr><td>' + list[a][0] + '</td><td>' + list[a][1] + '</td></tr>');
d.innerHTML = html.join('\n') + '</tbody>';
d.setAttribute('id', 'bbox-halp');
d.onclick = function () {
overlay.removeChild(d);
};
overlay.appendChild(d);
}
function keyDownHandler(e) {
if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
return;
var k = e.code + '', v = vid();
if (k == "ArrowLeft" || k == "KeyJ")
showPreviousImage();
else if (k == "ArrowRight" || k == "KeyL")
showNextImage();
else if (k == "Escape")
hideOverlay();
else if (k == "Home")
showFirstImage(e);
else if (k == "End")
showLastImage(e);
else if (k == "Space" || k == "KeyP" || k == "KeyK")
playpause();
else if (k == "KeyU" || k == "KeyO")
relseek(k == "KeyU" ? -10 : 10);
else if (k == "KeyM" && v) {
v.muted = vmute = !vmute;
mp_ctl();
} }
else if (k == "KeyR" && v) {
vloop = !vloop;
vnext = vnext && !vloop;
setVmode();
}
else if (k == "KeyC" && v) {
vnext = !vnext;
vloop = vloop && !vnext;
setVmode();
}
else if (k == "KeyF")
try {
if (isFullscreen)
document.exitFullscreen();
else
v.requestFullscreen();
}
catch (ex) { }
}
function setVmode() {
var v = vid();
ebi('bbox-vmode').style.display = v ? '' : 'none';
if (!v)
return;
var msg = 'When video ends, ', tts = '', lbl;
if (vloop) {
lbl = 'Loop';
msg += 'repeat it';
tts = '$NHotkey: R';
}
else if (vnext) {
lbl = 'Cont';
msg += 'continue to next';
tts = '$NHotkey: C';
}
else {
lbl = 'Stop';
msg += 'just stop'
}
btnVmode.setAttribute('aria-label', msg);
btnVmode.setAttribute('tt', msg + tts);
btnVmode.textContent = lbl;
v.loop = vloop
if (vloop && v.paused)
v.play();
}
function tglVmode() {
if (vloop) {
vnext = true;
vloop = false;
}
else if (vnext)
vnext = false;
else
vloop = true;
setVmode();
if (tt.en)
tt.show.bind(this)();
}
function keyUpHandler(e) {
if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
return;
var k = e.code + '';
if (k == "Space")
ev(e);
} }
var passiveSupp = false; var passiveSupp = false;
@@ -343,11 +248,9 @@ window.baguetteBox = (function () {
function bindEvents() { function bindEvents() {
bind(overlay, 'click', overlayClickHandler); bind(overlay, 'click', overlayClickHandler);
bind(btnPrev, 'click', showPreviousImage); bind(previousButton, 'click', showPreviousImage);
bind(btnNext, 'click', showNextImage); bind(nextButton, 'click', showNextImage);
bind(btnClose, 'click', hideOverlay); bind(closeButton, 'click', hideOverlay);
bind(btnVmode, 'click', tglVmode);
bind(btnHelp, 'click', halp);
bind(slider, 'contextmenu', contextmenuHandler); bind(slider, 'contextmenu', contextmenuHandler);
bind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent); bind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
bind(overlay, 'touchmove', touchmoveHandler, passiveEvent); bind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
@@ -357,11 +260,9 @@ window.baguetteBox = (function () {
function unbindEvents() { function unbindEvents() {
unbind(overlay, 'click', overlayClickHandler); unbind(overlay, 'click', overlayClickHandler);
unbind(btnPrev, 'click', showPreviousImage); unbind(previousButton, 'click', showPreviousImage);
unbind(btnNext, 'click', showNextImage); unbind(nextButton, 'click', showNextImage);
unbind(btnClose, 'click', hideOverlay); unbind(closeButton, 'click', hideOverlay);
unbind(btnVmode, 'click', tglVmode);
unbind(btnHelp, 'click', halp);
unbind(slider, 'contextmenu', contextmenuHandler); unbind(slider, 'contextmenu', contextmenuHandler);
unbind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent); unbind(overlay, 'touchstart', touchstartHandler, nonPassiveEvent);
unbind(overlay, 'touchmove', touchmoveHandler, passiveEvent); unbind(overlay, 'touchmove', touchmoveHandler, passiveEvent);
@@ -370,9 +271,9 @@ window.baguetteBox = (function () {
} }
function prepareOverlay(gallery, userOptions) { function prepareOverlay(gallery, userOptions) {
if (currentGallery === gallery) if (currentGallery === gallery) {
return; return;
}
currentGallery = gallery; currentGallery = gallery;
setOptions(userOptions); setOptions(userOptions);
slider.innerHTML = ''; slider.innerHTML = '';
@@ -386,8 +287,8 @@ window.baguetteBox = (function () {
fullImage.id = 'baguette-img-' + i; fullImage.id = 'baguette-img-' + i;
imagesElements.push(fullImage); imagesElements.push(fullImage);
imagesFiguresIds.push('bbox-figure-' + i); imagesFiguresIds.push('baguetteBox-figure-' + i);
imagesCaptionsIds.push('bbox-figcaption-' + i); imagesCaptionsIds.push('baguetteBox-figcaption-' + i);
slider.appendChild(imagesElements[i]); slider.appendChild(imagesElements[i]);
} }
overlay.setAttribute('aria-labelledby', imagesFiguresIds.join(' ')); overlay.setAttribute('aria-labelledby', imagesFiguresIds.join(' '));
@@ -395,21 +296,23 @@ window.baguetteBox = (function () {
} }
function setOptions(newOptions) { function setOptions(newOptions) {
if (!newOptions) if (!newOptions) {
newOptions = {}; newOptions = {};
}
for (var item in defaults) { for (var item in defaults) {
options[item] = defaults[item]; options[item] = defaults[item];
if (typeof newOptions[item] !== 'undefined') if (typeof newOptions[item] !== 'undefined') {
options[item] = newOptions[item]; options[item] = newOptions[item];
}
} }
slider.style.transition = (options.animation === 'fadeIn' ? 'opacity .4s ease' : slider.style.transition = (options.animation === 'fadeIn' ? 'opacity .4s ease' :
options.animation === 'slideIn' ? '' : 'none'); options.animation === 'slideIn' ? '' : 'none');
if (options.buttons === 'auto' && ('ontouchstart' in window || currentGallery.length === 1)) if (options.buttons === 'auto' && ('ontouchstart' in window || currentGallery.length === 1)) {
options.buttons = false; options.buttons = false;
}
btnPrev.style.display = btnNext.style.display = (options.buttons ? '' : 'none'); previousButton.style.display = nextButton.style.display = (options.buttons ? '' : 'none');
} }
function showOverlay(chosenImageIndex) { function showOverlay(chosenImageIndex) {
@@ -417,12 +320,11 @@ window.baguetteBox = (function () {
document.documentElement.style.overflowY = 'hidden'; document.documentElement.style.overflowY = 'hidden';
document.body.style.overflowY = 'scroll'; document.body.style.overflowY = 'scroll';
} }
if (overlay.style.display === 'block') if (overlay.style.display === 'block') {
return; return;
}
bind(document, 'keydown', keyDownHandler); bind(document, 'keydown', keyDownHandler);
bind(document, 'keyup', keyUpHandler);
bind(document, 'fullscreenchange', onFSC);
currentIndex = chosenImageIndex; currentIndex = chosenImageIndex;
touch = { touch = {
count: 0, count: 0,
@@ -439,48 +341,50 @@ window.baguetteBox = (function () {
// Fade in overlay // Fade in overlay
setTimeout(function () { setTimeout(function () {
overlay.className = 'visible'; overlay.className = 'visible';
if (options.bodyClass && document.body.classList) if (options.bodyClass && document.body.classList) {
document.body.classList.add(options.bodyClass); document.body.classList.add(options.bodyClass);
}
if (options.afterShow) if (options.afterShow) {
options.afterShow(); options.afterShow();
}
}, 50); }, 50);
if (options.onChange) {
if (options.onChange)
options.onChange(currentIndex, imagesElements.length); options.onChange(currentIndex, imagesElements.length);
}
documentLastFocus = document.activeElement; documentLastFocus = document.activeElement;
btnClose.focus(); initFocus();
isOverlayVisible = true; isOverlayVisible = true;
} }
function initFocus() {
if (options.buttons) {
previousButton.focus();
} else {
closeButton.focus();
}
}
function hideOverlay(e) { function hideOverlay(e) {
ev(e); ev(e);
playvid(false);
if (options.noScrollbars) { if (options.noScrollbars) {
document.documentElement.style.overflowY = 'auto'; document.documentElement.style.overflowY = 'auto';
document.body.style.overflowY = 'auto'; document.body.style.overflowY = 'auto';
} }
if (overlay.style.display === 'none') if (overlay.style.display === 'none') {
return; return;
}
unbind(document, 'keydown', keyDownHandler); unbind(document, 'keydown', keyDownHandler);
unbind(document, 'keyup', keyUpHandler);
unbind(document, 'fullscreenchange', onFSC);
// Fade out and hide the overlay // Fade out and hide the overlay
overlay.className = ''; overlay.className = '';
setTimeout(function () { setTimeout(function () {
overlay.style.display = 'none'; overlay.style.display = 'none';
if (options.bodyClass && document.body.classList) if (options.bodyClass && document.body.classList) {
document.body.classList.remove(options.bodyClass); document.body.classList.remove(options.bodyClass);
}
var h = ebi('bbox-halp'); if (options.afterHide) {
if (h)
h.parentNode.removeChild(h);
if (options.afterHide)
options.afterHide(); options.afterHide();
}
documentLastFocus && documentLastFocus.focus(); documentLastFocus && documentLastFocus.focus();
isOverlayVisible = false; isOverlayVisible = false;
}, 500); }, 500);
@@ -490,68 +394,59 @@ window.baguetteBox = (function () {
var imageContainer = imagesElements[index]; var imageContainer = imagesElements[index];
var galleryItem = currentGallery[index]; var galleryItem = currentGallery[index];
if (typeof imageContainer === 'undefined' || typeof galleryItem === 'undefined') if (typeof imageContainer === 'undefined' || typeof galleryItem === 'undefined') {
return; // out-of-bounds or gallery dirty return; // out-of-bounds or gallery dirty
}
if (imageContainer.querySelector('img, video')) if (imageContainer.getElementsByTagName('img')[0]) {
// was loaded, cb and bail // image is loaded, cb and bail
return callback ? callback() : null; if (callback) {
callback();
// maybe unloaded video }
while (imageContainer.firstChild) return;
imageContainer.removeChild(imageContainer.firstChild); }
var imageElement = galleryItem.imageElement, var imageElement = galleryItem.imageElement,
imageSrc = imageElement.href, imageSrc = imageElement.href,
is_vid = re_v.test(imageSrc), thumbnailElement = imageElement.getElementsByTagName('img')[0],
thumbnailElement = imageElement.querySelector('img, video'),
imageCaption = typeof options.captions === 'function' ? imageCaption = typeof options.captions === 'function' ?
options.captions.call(currentGallery, imageElement) : options.captions.call(currentGallery, imageElement) :
imageElement.getAttribute('data-caption') || imageElement.title; imageElement.getAttribute('data-caption') || imageElement.title;
imageSrc += imageSrc.indexOf('?') < 0 ? '?cache' : '&cache';
if (is_vid && index != currentIndex)
return; // no preload
var figure = mknod('figure'); var figure = mknod('figure');
figure.id = 'bbox-figure-' + index; figure.id = 'baguetteBox-figure-' + index;
figure.innerHTML = '<div class="bbox-spinner">' + figure.innerHTML = '<div class="baguetteBox-spinner">' +
'<div class="bbox-double-bounce1"></div>' + '<div class="baguetteBox-double-bounce1"></div>' +
'<div class="bbox-double-bounce2"></div>' + '<div class="baguetteBox-double-bounce2"></div>' +
'</div>'; '</div>';
if (options.captions && imageCaption) { if (options.captions && imageCaption) {
var figcaption = mknod('figcaption'); var figcaption = mknod('figcaption');
figcaption.id = 'bbox-figcaption-' + index; figcaption.id = 'baguetteBox-figcaption-' + index;
figcaption.innerHTML = imageCaption; figcaption.innerHTML = imageCaption;
figure.appendChild(figcaption); figure.appendChild(figcaption);
} }
imageContainer.appendChild(figure); imageContainer.appendChild(figure);
var image = mknod(is_vid ? 'video' : 'img'); var image = mknod('img');
clmod(imageContainer, 'vid', is_vid); image.onload = function () {
image.addEventListener(is_vid ? 'loadedmetadata' : 'load', function () {
// Remove loader element // Remove loader element
var spinner = QS('#baguette-img-' + index + ' .bbox-spinner'); var spinner = document.querySelector('#baguette-img-' + index + ' .baguetteBox-spinner');
figure.removeChild(spinner); figure.removeChild(spinner);
if (!options.async && callback) if (!options.async && callback) {
callback(); callback();
}); }
};
image.setAttribute('src', imageSrc); image.setAttribute('src', imageSrc);
if (is_vid) {
image.setAttribute('controls', 'controls');
image.onended = vidEnd;
}
image.alt = thumbnailElement ? thumbnailElement.alt || '' : ''; image.alt = thumbnailElement ? thumbnailElement.alt || '' : '';
if (options.titleTag && imageCaption) if (options.titleTag && imageCaption) {
image.title = imageCaption; image.title = imageCaption;
}
figure.appendChild(image); figure.appendChild(image);
if (options.async && callback) if (options.async && callback) {
callback(); callback();
}
} }
function showNextImage(e) { function showNextImage(e) {
@@ -564,20 +459,26 @@ window.baguetteBox = (function () {
return show(currentIndex - 1); return show(currentIndex - 1);
} }
function showFirstImage(e) { function showFirstImage(event) {
if (e) if (event) {
e.preventDefault(); event.preventDefault();
}
return show(0); return show(0);
} }
function showLastImage(e) { function showLastImage(event) {
if (e) if (event) {
e.preventDefault(); event.preventDefault();
}
return show(currentGallery.length - 1); return show(currentGallery.length - 1);
} }
/**
* Move the gallery to a specific index
* @param `index` {number} - the position of the image
* @param `gallery` {array} - gallery which should be opened, if omitted assumes the currently opened one
* @return {boolean} - true on success or false if the index is invalid
*/
function show(index, gallery) { function show(index, gallery) {
if (!isOverlayVisible && index >= 0 && index < gallery.length) { if (!isOverlayVisible && index >= 0 && index < gallery.length) {
prepareOverlay(gallery, options); prepareOverlay(gallery, options);
@@ -585,25 +486,18 @@ window.baguetteBox = (function () {
return true; return true;
} }
if (index < 0) { if (index < 0) {
if (options.animation) if (options.animation) {
bounceAnimation('left'); bounceAnimation('left');
}
return false; return false;
} }
if (index >= imagesElements.length) { if (index >= imagesElements.length) {
if (options.animation) if (options.animation) {
bounceAnimation('right'); bounceAnimation('right');
}
return false; return false;
} }
var v = vid();
if (v) {
v.src = '';
v.load();
v.parentNode.removeChild(v);
}
currentIndex = index; currentIndex = index;
loadImage(currentIndex, function () { loadImage(currentIndex, function () {
preloadNext(currentIndex); preloadNext(currentIndex);
@@ -611,49 +505,17 @@ window.baguetteBox = (function () {
}); });
updateOffset(); updateOffset();
if (options.onChange) if (options.onChange) {
options.onChange(currentIndex, imagesElements.length); options.onChange(currentIndex, imagesElements.length);
}
return true; return true;
} }
function vid() { /**
return imagesElements[currentIndex].querySelector('video'); * Triggers the bounce animation
} * @param {('left'|'right')} direction - Direction of the movement
*/
function playvid(play) {
if (vid())
vid()[play ? 'play' : 'pause']();
}
function playpause() {
var v = vid();
if (v)
v[v.paused ? "play" : "pause"]();
}
function relseek(sec) {
if (vid())
vid().currentTime += sec;
}
function vidEnd() {
if (this == vid() && vnext)
showNextImage();
}
function mp_ctl() {
var v = vid();
if (!vmute && v && mp.au && !mp.au.paused) {
mp.fade_out();
resume_mp = true;
}
else if (resume_mp && (vmute || !v) && mp.au && mp.au.paused) {
mp.fade_in();
resume_mp = false;
}
}
function bounceAnimation(direction) { function bounceAnimation(direction) {
slider.className = 'bounce-from-' + direction; slider.className = 'bounce-from-' + direction;
setTimeout(function () { setTimeout(function () {
@@ -672,30 +534,21 @@ window.baguetteBox = (function () {
} else { } else {
slider.style.transform = 'translate3d(' + offset + ',0,0)'; slider.style.transform = 'translate3d(' + offset + ',0,0)';
} }
playvid(false);
var v = vid();
if (v) {
playvid(true);
v.muted = vmute;
v.loop = vloop;
}
mp_ctl();
setVmode();
} }
function preloadNext(index) { function preloadNext(index) {
if (index - currentIndex >= options.preload) if (index - currentIndex >= options.preload) {
return; return;
}
loadImage(index + 1, function () { loadImage(index + 1, function () {
preloadNext(index + 1); preloadNext(index + 1);
}); });
} }
function preloadPrev(index) { function preloadPrev(index) {
if (currentIndex - index >= options.preload) if (currentIndex - index >= options.preload) {
return; return;
}
loadImage(index - 1, function () { loadImage(index - 1, function () {
preloadPrev(index - 1); preloadPrev(index - 1);
}); });
@@ -713,8 +566,7 @@ window.baguetteBox = (function () {
unbindEvents(); unbindEvents();
clearCachedData(); clearCachedData();
unbind(document, 'keydown', keyDownHandler); unbind(document, 'keydown', keyDownHandler);
unbind(document, 'keyup', keyUpHandler); document.getElementsByTagName('body')[0].removeChild(ebi('baguetteBox-overlay'));
document.getElementsByTagName('body')[0].removeChild(ebi('bbox-overlay'));
data = {}; data = {};
currentGallery = []; currentGallery = [];
currentIndex = 0; currentIndex = 0;
@@ -725,8 +577,6 @@ window.baguetteBox = (function () {
show: show, show: show,
showNext: showNextImage, showNext: showNextImage,
showPrevious: showPreviousImage, showPrevious: showPreviousImage,
relseek: relseek,
playpause: playpause,
hide: hideOverlay, hide: hideOverlay,
destroy: destroyPlugin destroy: destroyPlugin
}; };

View File

@@ -25,121 +25,34 @@ html, body {
body { body {
padding-bottom: 5em; padding-bottom: 5em;
} }
pre, code, tt { #tt {
font-family: monospace, monospace;
}
#tt, #toast {
position: fixed; position: fixed;
max-width: 34em; max-width: 34em;
background: #222; background: #222;
border: 0 solid #777; border: 0 solid #555;
overflow: hidden;
margin-top: 1em;
padding: 0 1em;
height: 0;
opacity: .1;
transition: opacity 0.14s, height 0.14s, padding 0.14s;
box-shadow: 0 .2em .5em #222; box-shadow: 0 .2em .5em #222;
border-radius: .4em; border-radius: .4em;
z-index: 9001; z-index: 9001;
} }
#tt {
overflow: hidden;
margin-top: 1em;
padding: 0 1.3em;
height: 0;
opacity: .1;
transition: opacity 0.14s, height 0.14s, padding 0.14s;
}
#toast {
top: 1.4em;
right: -1em;
line-height: 1.5em;
padding: 1em 1.3em;
border-width: .4em 0;
transform: translateX(100%);
transition:
transform .4s cubic-bezier(.2, 1.2, .5, 1),
right .4s cubic-bezier(.2, 1.2, .5, 1);
text-shadow: 1px 1px 0 #000;
color: #fff;
}
#toastc {
display: inline-block;
position: absolute;
overflow: hidden;
left: 0;
width: 0;
opacity: 0;
padding: .3em 0;
margin: -.3em 0 0 0;
line-height: 1.5em;
color: #000;
border: none;
outline: none;
text-shadow: none;
border-radius: .5em 0 0 .5em;
transition: left .3s, width .3s, padding .3s, opacity .3s;
}
#toast.vis {
right: 1.3em;
transform: unset;
}
#toast.vis #toastc {
left: -2em;
width: .4em;
padding: .3em .8em;
opacity: 1;
}
#toast.inf {
background: #07a;
border-color: #0be;
}
#toast.inf #toastc {
background: #0be;
}
#toast.ok {
background: #4a0;
border-color: #8e4;
}
#toast.ok #toastc {
background: #8e4;
}
#toast.warn {
background: #970;
border-color: #fc0;
}
#toast.warn #toastc {
background: #fc0;
}
#toast.err {
background: #900;
border-color: #d06;
}
#toast.err #toastc {
background: #d06;
}
#tt.b {
padding: 0 2em;
border-radius: .5em;
box-shadow: 0 .2em 1em #000;
}
#tt.show { #tt.show {
padding: 1em 1.3em; padding: 1em;
border-width: .4em 0;
height: auto; height: auto;
border-width: .2em 0;
opacity: 1; opacity: 1;
} }
#tt.show.b {
padding: 1.5em 2em;
border-width: .5em 0;
}
#tt code { #tt code {
background: #3c3c3c; background: #3c3c3c;
padding: .1em .3em; padding: .2em .3em;
border-top: 1px solid #777; border-top: 1px solid #777;
border-radius: .3em; border-radius: .3em;
line-height: 1.7em; font-family: monospace, monospace;
} line-height: 2em;
#tt em {
color: #f6a;
} }
#path, #path,
#path * { #path * {
@@ -171,10 +84,6 @@ pre, code, tt {
padding: .3em 0; padding: .3em 0;
scroll-margin-top: 45vh; scroll-margin-top: 45vh;
} }
#files tr {
scroll-margin-top: 25vh;
scroll-margin-bottom: 20vh;
}
#files tbody div a { #files tbody div a {
color: #f5a; color: #f5a;
} }
@@ -229,7 +138,8 @@ a, #files tbody div a:last-child {
border-top: 1px solid #383838; border-top: 1px solid #383838;
} }
#files tbody td:nth-child(3) { #files tbody td:nth-child(3) {
font-family: monospace, monospace; font-family: monospace;
font-size: 1.3em;
text-align: right; text-align: right;
padding-right: 1em; padding-right: 1em;
white-space: nowrap; white-space: nowrap;
@@ -289,31 +199,15 @@ a, #files tbody div a:last-child {
margin: .8em 0; margin: .8em 0;
} }
#srv_info { #srv_info {
color: #a73; opacity: .5;
background: #333;
position: absolute;
font-size: .8em; font-size: .8em;
top: .5em; color: #fc5;
position: absolute;
top: .5em;
left: 2em; left: 2em;
padding-right: .5em;
} }
#srv_info span { #srv_info span {
color: #aaa; color: #fff;
}
#acc_info {
position: absolute;
font-size: .81em;
top: .5em;
right: 2em;
color: #999;
}
#acc_info span {
color: #999;
margin-right: .6em;
}
#acc_info span.warn {
color: #f4c;
border-bottom: 1px solid rgba(255,68,204,0.6);
} }
#files tbody a.play { #files tbody a.play {
color: #e70; color: #e70;
@@ -340,7 +234,6 @@ html.light #ggrid a.sel {
border-color: #c37; border-color: #c37;
} }
#files tbody tr.sel:hover td, #files tbody tr.sel:hover td,
#files tbody tr.sel:focus td,
#ggrid a.sel:hover, #ggrid a.sel:hover,
html.light #ggrid a.sel:hover { html.light #ggrid a.sel:hover {
color: #fff; color: #fff;
@@ -375,21 +268,6 @@ html.light #ggrid a.sel {
color: #fff; color: #fff;
text-shadow: 0 0 1px #fff; text-shadow: 0 0 1px #fff;
} }
#files tr:focus {
outline: none;
position: relative;
}
#files tr:focus td {
background: #111;
border-color: #fc0 #111 #fc0 #111;
box-shadow: 0 .2em 0 #fc0, 0 -.2em 0 #fc0;
}
#files tr:focus td:first-child {
box-shadow: -.2em .2em 0 #fc0, -.2em -.2em 0 #fc0;
}
#files tr:focus+tr td {
border-top: 1px solid transparent;
}
#blocked { #blocked {
position: fixed; position: fixed;
top: 0; top: 0;
@@ -447,18 +325,10 @@ html.light #ggrid a.sel {
height: 100%; height: 100%;
background: #3c3c3c; background: #3c3c3c;
} }
#wtgrid,
#wtico { #wtico {
cursor: url(/.cpr/dd/4.png), pointer; cursor: url(/.cpr/dd/4.png), pointer;
animation: cursor 500ms; animation: cursor 500ms;
position: relative;
top: -.06em;
} }
#wtgrid {
font-size: .8em;
top: -.12em;
}
#wtgrid:hover,
#wtico:hover { #wtico:hover {
animation: cursor 500ms infinite; animation: cursor 500ms infinite;
} }
@@ -474,9 +344,9 @@ html.light #ggrid a.sel {
} }
#wtoggle { #wtoggle {
position: absolute; position: absolute;
white-space: nowrap;
top: -1.2em; top: -1.2em;
right: 0; right: 0;
width: 1.2em;
height: 1em; height: 1em;
font-size: 2em; font-size: 2em;
line-height: 1em; line-height: 1em;
@@ -485,7 +355,7 @@ html.light #ggrid a.sel {
background: #3c3c3c; background: #3c3c3c;
box-shadow: 0 0 .5em #222; box-shadow: 0 0 .5em #222;
border-radius: .3em 0 0 0; border-radius: .3em 0 0 0;
padding: .2em .2em; padding: .2em 0 0 .07em;
color: #fff; color: #fff;
} }
#wzip, #wnp { #wzip, #wnp {
@@ -507,6 +377,12 @@ html.light #ggrid a.sel {
#wtoggle * { #wtoggle * {
line-height: 1em; line-height: 1em;
} }
#wtoggle.np {
width: 5.5em;
}
#wtoggle.sel {
width: 6.4em;
}
#wtoggle.sel #wzip, #wtoggle.sel #wzip,
#wtoggle.np #wnp { #wtoggle.np #wnp {
display: inline-block; display: inline-block;
@@ -514,42 +390,15 @@ html.light #ggrid a.sel {
#wtoggle.sel.np #wnp { #wtoggle.sel.np #wnp {
display: none; display: none;
} }
#wfm a,
#wzip a { #wzip a {
font-size: .5em; font-size: .4em;
padding: 0 .3em; padding: 0 .3em;
margin: -.3em .2em; margin: -.3em .2em;
position: relative; position: relative;
display: inline-block; display: inline-block;
} }
#wfm span { #wzip a+a {
font-size: .6em; margin-left: .8em;
display: block;
}
#wfm a:not(.en) {
opacity: .3;
color: #f6c;
}
html.light #wfm a:not(.en) {
color: #c4a;
}
#files tbody tr.c1 td {
animation: fcut1 .5s ease-out;
}
#files tbody tr.c2 td {
animation: fcut2 .5s ease-out;
}
@keyframes fcut1 {
0% {opacity:0}
100% {opacity:1}
}
@keyframes fcut2 {
0% {opacity:0}
100% {opacity:1}
}
#wzip a {
font-size: .4em;
margin: -.3em .3em;
} }
#wtoggle.sel #wzip #selzip { #wtoggle.sel #wzip #selzip {
top: -.6em; top: -.6em;
@@ -615,17 +464,6 @@ html.light #wfm a:not(.en) {
max-width: 9em; max-width: 9em;
} }
} }
@media (max-width: 35em) {
#ops>a[data-dest="new_md"],
#ops>a[data-dest="msg"] {
display: none;
}
#op_mkdir.act+div,
#op_mkdir.act+div+div {
display: block;
margin-top: 1em;
}
}
@@ -923,14 +761,9 @@ input.eq_gain {
display: block; display: block;
width: 1em; width: 1em;
border-radius: .2em; border-radius: .2em;
margin: -1.2em auto 0 auto; margin: -1.3em auto 0 auto;
top: 2em;
position: relative;
background: #444; background: #444;
} }
#files th span {
position: relative;
}
#files>thead>tr>th.min, #files>thead>tr>th.min,
#files td.min { #files td.min {
display: none; display: none;
@@ -979,13 +812,11 @@ input.eq_gain {
border-bottom: 1px solid #555; border-bottom: 1px solid #555;
} }
#thumbs, #thumbs,
#au_osd_cv, #au_osd_cv {
#u2tdate {
opacity: .3; opacity: .3;
} }
#griden.on+#thumbs, #griden.on+#thumbs,
#au_os_ctl.on+#au_osd_cv, #au_os_ctl.on+#au_osd_cv {
#u2turbo.on+#u2tdate {
opacity: 1; opacity: 1;
} }
#ghead { #ghead {
@@ -1057,8 +888,7 @@ html.light #ggrid a:hover {
#pvol, #pvol,
#barbuf, #barbuf,
#barpos, #barpos,
#u2conf label, #u2conf label {
#ops {
-webkit-user-select: none; -webkit-user-select: none;
-moz-user-select: none; -moz-user-select: none;
-ms-user-select: none; -ms-user-select: none;
@@ -1091,19 +921,13 @@ html.light {
} }
html.light #tt { html.light #tt {
background: #fff; background: #fff;
border-color: #888 #000 #777 #000; border-color: #888;
}
html.light #tt,
html.light #toast {
box-shadow: 0 .3em 1em rgba(0,0,0,0.4); box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
} }
html.light #tt code { html.light #tt code {
background: #060; background: #060;
color: #fff; color: #fff;
} }
html.light #tt em {
color: #d38;
}
html.light #ops, html.light #ops,
html.light .opbox, html.light .opbox,
html.light #srch_form { html.light #srch_form {
@@ -1134,14 +958,10 @@ html.light .tgl.btn.on {
} }
html.light #srv_info { html.light #srv_info {
color: #c83; color: #c83;
background: #eee;
}
html.light #srv_info,
html.light #acc_info {
text-shadow: 1px 1px 0 #fff; text-shadow: 1px 1px 0 #fff;
} }
html.light #srv_info span { html.light #srv_info span {
color: #777; color: #000;
} }
html.light #treeul a+a { html.light #treeul a+a {
background: inherit; background: inherit;
@@ -1188,17 +1008,6 @@ html.light #files td {
html.light #files tbody tr:last-child td { html.light #files tbody tr:last-child td {
border-bottom: .2em solid #ccc; border-bottom: .2em solid #ccc;
} }
html.light #files tr:focus td {
background: #fff;
border-color: #c37;
box-shadow: 0 .2em 0 #e80 , 0 -.2em 0 #e80;
}
html.light #files tr:focus td:first-child {
box-shadow: -.2em .2em 0 #e80, -.2em -.2em 0 #e80;
}
html.light #files tr.sel td {
background: #925;
}
html.light #files td:nth-child(2n) { html.light #files td:nth-child(2n) {
color: #d38; color: #d38;
} }
@@ -1252,8 +1061,7 @@ html.light #wnp {
html.light #barbuf { html.light #barbuf {
background: none; background: none;
} }
html.light #files tr.sel:hover td, html.light #files tr.sel:hover td {
html.light #files tr.sel:focus td {
background: #c37; background: #c37;
} }
html.light #files tr.sel td { html.light #files tr.sel td {
@@ -1320,7 +1128,7 @@ html.light #tree::-webkit-scrollbar {
#bbox-overlay { #baguetteBox-overlay {
display: none; display: none;
opacity: 0; opacity: 0;
position: fixed; position: fixed;
@@ -1330,66 +1138,58 @@ html.light #tree::-webkit-scrollbar {
left: 0; left: 0;
width: 100%; width: 100%;
height: 100%; height: 100%;
z-index: 10; z-index: 1000000;
background: rgba(0, 0, 0, 0.8); background: rgba(0, 0, 0, 0.8);
transition: opacity .3s ease; transition: opacity .3s ease;
} }
#bbox-overlay.visible { #baguetteBox-overlay.visible {
opacity: 1; opacity: 1;
} }
.full-image { #baguetteBox-overlay .full-image {
display: inline-block; display: inline-block;
position: relative; position: relative;
width: 100%; width: 100%;
height: 100%; height: 100%;
text-align: center; text-align: center;
} }
.full-image figure { #baguetteBox-overlay .full-image figure {
display: inline; display: inline;
margin: 0; margin: 0;
height: 100%; height: 100%;
} }
.full-image img, #baguetteBox-overlay .full-image img {
.full-image video {
display: inline-block; display: inline-block;
width: auto; width: auto;
height: auto; height: auto;
max-width: 100%;
max-height: 100%; max-height: 100%;
max-height: calc(100% - 1.4em); max-width: 100%;
margin-bottom: 1.4em;
vertical-align: middle; vertical-align: middle;
box-shadow: 0 0 8px rgba(0, 0, 0, 0.6); box-shadow: 0 0 8px rgba(0, 0, 0, 0.6);
} }
.full-image video { #baguetteBox-overlay .full-image figcaption {
background: #333;
}
.full-image figcaption {
display: block; display: block;
position: fixed; position: absolute;
bottom: .1em; bottom: 0;
width: 100%; width: 100%;
text-align: center; text-align: center;
line-height: 1.8;
white-space: normal; white-space: normal;
color: #ccc; color: #ccc;
} }
#bbox-overlay figcaption a { #baguetteBox-overlay figcaption a {
background: rgba(0, 0, 0, 0.6); background: rgba(0, 0, 0, 0.6);
border-radius: .4em; border-radius: .4em;
padding: .3em .6em; padding: .3em .6em;
} }
html.light #bbox-overlay figcaption a { #baguetteBox-overlay .full-image:before {
color: #0bf;
}
.full-image:before {
content: ""; content: "";
display: inline-block; display: inline-block;
height: 50%; height: 50%;
width: 1px; width: 1px;
margin-right: -1px; margin-right: -1px;
} }
#bbox-slider { #baguetteBox-slider {
position: fixed; position: absolute;
left: 0; left: 0;
top: 0; top: 0;
height: 100%; height: 100%;
@@ -1397,10 +1197,10 @@ html.light #bbox-overlay figcaption a {
white-space: nowrap; white-space: nowrap;
transition: left .2s ease, transform .2s ease; transition: left .2s ease, transform .2s ease;
} }
.bounce-from-right { #baguetteBox-slider.bounce-from-right {
animation: bounceFromRight .4s ease-out; animation: bounceFromRight .4s ease-out;
} }
.bounce-from-left { #baguetteBox-slider.bounce-from-left {
animation: bounceFromLeft .4s ease-out; animation: bounceFromLeft .4s ease-out;
} }
@keyframes bounceFromRight { @keyframes bounceFromRight {
@@ -1413,63 +1213,48 @@ html.light #bbox-overlay figcaption a {
50% {margin-left: 30px} 50% {margin-left: 30px}
100% {margin-left: 0} 100% {margin-left: 0}
} }
#bbox-next, .baguetteBox-button#next-button,
#bbox-prev { .baguetteBox-button#previous-button {
top: 50%; top: 50%;
top: calc(50% - 30px); top: calc(50% - 30px);
width: 44px; width: 44px;
height: 60px; height: 60px;
} }
.bbox-btn { .baguetteBox-button {
position: fixed; position: absolute;
}
#bbox-overlay button {
cursor: pointer; cursor: pointer;
outline: none; outline: none;
padding: 0 .3em; padding: 0;
margin: 0 .4em; margin: 0;
border: 0; border: 0;
border-radius: 15%; border-radius: 15%;
background: rgba(50, 50, 50, 0.5); background: rgba(50, 50, 50, 0.5);
color: rgba(255,255,255,0.7); color: #ddd;
font: 1.6em sans-serif;
transition: background-color .3s ease; transition: background-color .3s ease;
transition: color .3s ease;
font-size: 1.4em;
line-height: 1.4em;
vertical-align: top;
} }
#bbox-overlay button:focus, .baguetteBox-button:focus,
#bbox-overlay button:hover { .baguetteBox-button:hover {
color: rgba(255,255,255,0.9);
background: rgba(50, 50, 50, 0.9); background: rgba(50, 50, 50, 0.9);
} }
#bbox-next { #next-button {
right: 1%;
}
#bbox-prev {
left: 1%;
}
#bbox-btns {
top: .5em;
right: 2%; right: 2%;
position: fixed;
} }
#bbox-halp { #previous-button {
color: #fff; left: 2%;
background: #333; }
#close-button {
top: 20px;
right: 2%;
width: 30px;
height: 30px;
}
.baguetteBox-button svg {
position: absolute; position: absolute;
top: 0;
left: 0; left: 0;
z-index: 20; top: 0;
padding: .4em;
} }
#bbox-halp td { .baguetteBox-spinner {
padding: .2em .5em;
}
#bbox-halp td:first-child {
text-align: right;
}
.bbox-spinner {
width: 40px; width: 40px;
height: 40px; height: 40px;
display: inline-block; display: inline-block;
@@ -1479,8 +1264,8 @@ html.light #bbox-overlay figcaption a {
margin-top: -20px; margin-top: -20px;
margin-left: -20px; margin-left: -20px;
} }
.bbox-double-bounce1, .baguetteBox-double-bounce1,
.bbox-double-bounce2 { .baguetteBox-double-bounce2 {
width: 100%; width: 100%;
height: 100%; height: 100%;
border-radius: 50%; border-radius: 50%;
@@ -1491,7 +1276,7 @@ html.light #bbox-overlay figcaption a {
left: 0; left: 0;
animation: bounce 2s infinite ease-in-out; animation: bounce 2s infinite ease-in-out;
} }
.bbox-double-bounce2 { .baguetteBox-double-bounce2 {
animation-delay: -1s; animation-delay: -1s;
} }
@keyframes bounce { @keyframes bounce {

View File

@@ -39,23 +39,23 @@
<div id="op_mkdir" class="opview opbox act"> <div id="op_mkdir" class="opview opbox act">
<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}"> <form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
<input type="hidden" name="act" value="mkdir" /> <input type="hidden" name="act" value="mkdir" />
📂<input type="text" name="name" size="30"> <input type="text" name="name" size="30">
<input type="submit" value="make directory"> <input type="submit" value="mkdir">
</form> </form>
</div> </div>
<div id="op_new_md" class="opview opbox"> <div id="op_new_md" class="opview opbox">
<form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}"> <form method="post" enctype="multipart/form-data" accept-charset="utf-8" action="{{ url_suf }}">
<input type="hidden" name="act" value="new_md" /> <input type="hidden" name="act" value="new_md" />
📝<input type="text" name="name" size="30"> <input type="text" name="name" size="30">
<input type="submit" value="new markdown doc"> <input type="submit" value="create doc">
</form> </form>
</div> </div>
<div id="op_msg" class="opview opbox act"> <div id="op_msg" class="opview opbox act">
<form method="post" enctype="application/x-www-form-urlencoded" accept-charset="utf-8" action="{{ url_suf }}"> <form method="post" enctype="application/x-www-form-urlencoded" accept-charset="utf-8" action="{{ url_suf }}">
📟<input type="text" name="msg" size="30"> <input type="text" name="msg" size="30">
<input type="submit" value="send msg to server log"> <input type="submit" value="send msg">
</form> </form>
</div> </div>
@@ -64,7 +64,7 @@
<div id="op_cfg" class="opview opbox opwide"></div> <div id="op_cfg" class="opview opbox opwide"></div>
<h1 id="path"> <h1 id="path">
<a href="#" id="entree" tt="show navpane (directory tree sidebar)$NHotkey: B">🌲</a> <a href="#" id="entree" tt="show directory tree$NHotkey: B">🌲</a>
{%- for n in vpnodes %} {%- for n in vpnodes %}
<a href="/{{ n[0] }}">{{ n[1] }}</a> <a href="/{{ n[0] }}">{{ n[1] }}</a>
{%- endfor %} {%- endfor %}
@@ -121,13 +121,10 @@
<div id="widget"></div> <div id="widget"></div>
<script> <script>
var acct = "{{ acct }}", var perms = {{ perms }},
perms = {{ perms }},
tag_order_cfg = {{ tag_order }}, tag_order_cfg = {{ tag_order }},
have_up2k_idx = {{ have_up2k_idx|tojson }}, have_up2k_idx = {{ have_up2k_idx|tojson }},
have_tags_idx = {{ have_tags_idx|tojson }}, have_tags_idx = {{ have_tags_idx|tojson }},
have_mv = {{ have_mv|tojson }},
have_del = {{ have_del|tojson }},
have_zip = {{ have_zip|tojson }}; have_zip = {{ have_zip|tojson }};
</script> </script>
<script src="/.cpr/util.js?_={{ ts }}"></script> <script src="/.cpr/util.js?_={{ ts }}"></script>

View File

@@ -29,20 +29,14 @@ ebi('ops').innerHTML = (
// media player // media player
ebi('widget').innerHTML = ( ebi('widget').innerHTML = (
'<div id="wtoggle">' + '<div id="wtoggle">' +
'<span id="wfm"><a' + '<span id="wzip"><a' +
' href="#" id="fren" tt="rename selected item$NHotkey: F2">✎<span>name</span></a><a' + ' href="#" id="selall" tt="select all files">sel.<br />all</a><a' +
' href="#" id="fdel" tt="delete selected items$NHotkey: ctrl-K">⌫<span>delete</span></a><a' +
' href="#" id="fcut" tt="cut selected items &lt;small&gt;(then paste somewhere else)&lt;/small&gt;$NHotkey: ctrl-X">✂<span>cut</span></a><a' +
' href="#" id="fpst" tt="paste a previously cut/copied selection$NHotkey: ctrl-V">📋<span>paste</span></a>' +
'</span><span id="wzip"><a' +
' href="#" id="selall" tt="select all files$NHotkey: ctrl-A (when file focused)">sel.<br />all</a><a' +
' href="#" id="selinv" tt="invert selection">sel.<br />inv.</a><a' + ' href="#" id="selinv" tt="invert selection">sel.<br />inv.</a><a' +
' href="#" id="selzip" tt="download selection as archive">zip</a>' + ' href="#" id="selzip" tt="download selection as archive">zip</a>' +
'</span><span id="wnp"><a' + '</span><span id="wnp"><a' +
' href="#" id="npirc" tt="copy irc-formatted track info">📋irc</a><a' + ' href="#" id="npirc" tt="copy irc-formatted track info">📋irc</a><a' +
' href="#" id="nptxt" tt="copy plaintext track info">📋txt</a>' + ' href="#" id="nptxt" tt="copy plaintext track info">📋txt</a>' +
'</span><a' + '</span><a' +
' href="#" id="wtgrid" tt="toggle grid/list view">田</a><a' +
' href="#" id="wtico">♫</a>' + ' href="#" id="wtico">♫</a>' +
'</div>' + '</div>' +
'<div id="widgeti">' + '<div id="widgeti">' +
@@ -67,7 +61,7 @@ ebi('op_up2k').innerHTML = (
' </td>\n' + ' </td>\n' +
' <td rowspan="2">\n' + ' <td rowspan="2">\n' +
' <input type="checkbox" id="ask_up" />\n' + ' <input type="checkbox" id="ask_up" />\n' +
' <label for="ask_up" tt="ask for confirmation before upload starts">💭</label>\n' + ' <label for="ask_up" tt="ask for confirmation befofre upload starts">💭</label>\n' +
' </td>\n' + ' </td>\n' +
' <td rowspan="2">\n' + ' <td rowspan="2">\n' +
' <input type="checkbox" id="flag_en" />\n' + ' <input type="checkbox" id="flag_en" />\n' +
@@ -139,24 +133,17 @@ ebi('op_cfg').innerHTML = (
(have_zip ? ( (have_zip ? (
'<div><h3>folder download</h3><div id="arc_fmt"></div></div>\n' '<div><h3>folder download</h3><div id="arc_fmt"></div></div>\n'
) : '') + ) : '') +
'<div>\n' +
' <h3>up2k switches</h3>\n' +
' <div>\n' +
' <a id="u2turbo" class="tgl btn ttb" href="#" tt="the yolo button, you probably DO NOT want to enable this:$N$Nuse this if you were uploading a huge amount of files and had to restart for some reason, and want to continue the upload ASAP$N$Nthis replaces the hash-check with a simple <em>&quot;does this have the same filesize on the server?&quot;</em> so if the file contents are different it will NOT be uploaded$N$Nyou should turn this off when the upload is done, and then &quot;upload&quot; the same files again to let the client verify them">turbo</a>\n' +
' <a id="u2tdate" class="tgl btn ttb" href="#" tt="has no effect unless the turbo button is enabled$N$Nreduces the yolo factor by a tiny amount; checks whether the file timestamps on the server matches yours$N$Nshould <em>theoretically</em> catch most unfinished/corrupted uploads, but is not a substitute for doing a verification pass with turbo disabled afterwards">date-chk</a>\n' +
' </div>\n' +
'</div>\n' +
'<div><h3>key notation</h3><div id="key_notation"></div></div>\n' + '<div><h3>key notation</h3><div id="key_notation"></div></div>\n' +
'<div class="fill"><h3>hidden columns</h3><div id="hcols"></div></div>' '<div class="fill"><h3>hidden columns</h3><div id="hcols"></div></div>'
); );
// navpane // tree sidebar
ebi('tree').innerHTML = ( ebi('tree').innerHTML = (
'<div id="treeh">\n' + '<div id="treeh">\n' +
' <a href="#" id="detree" tt="show breadcrumbs$NHotkey: B">🍞...</a>\n' + ' <a href="#" id="detree" tt="show breadcrumbs$NHotkey: B">🍞...</a>\n' +
' <a href="#" class="btn" step="2" id="twobytwo" tt="Hotkey: A">+</a>\n' + ' <a href="#" class="btn" step="2" id="twobytwo">+</a>\n' +
' <a href="#" class="btn" step="-2" id="twig" tt="Hotkey: D">&ndash;</a>\n' + ' <a href="#" class="btn" step="-2" id="twig">&ndash;</a>\n' +
' <a href="#" class="tgl btn" id="dyntree" tt="autogrow as tree expands">a</a>\n' + ' <a href="#" class="tgl btn" id="dyntree" tt="autogrow as tree expands">a</a>\n' +
'</div>\n' + '</div>\n' +
'<ul id="treeul"></ul>\n' + '<ul id="treeul"></ul>\n' +
@@ -173,19 +160,16 @@ ebi('tree').innerHTML = (
function opclick(e) { function opclick(e) {
var dest = this.getAttribute('data-dest');
swrite('opmode', dest || null);
if (ctrl(e))
return;
ev(e); ev(e);
var dest = this.getAttribute('data-dest');
goto(dest); goto(dest);
swrite('opmode', dest || null);
var input = QS('.opview.act input:not([type="hidden"])') var input = QS('.opview.act input:not([type="hidden"])')
if (input && !is_touch) { if (input && !is_touch)
tt.skip = true;
input.focus(); input.focus();
}
} }
@@ -285,7 +269,7 @@ var mpl = (function () {
r.os_ctl = !r.os_ctl && have_mctl; r.os_ctl = !r.os_ctl && have_mctl;
bcfg_set('au_os_ctl', r.os_ctl); bcfg_set('au_os_ctl', r.os_ctl);
if (!have_mctl) if (!have_mctl)
toast.err(5, 'need firefox 82+ or chrome 73+'); alert('need firefox 82+ or chrome 73+');
}; };
ebi('au_osd_cv').onclick = function (e) { ebi('au_osd_cv').onclick = function (e) {
@@ -613,7 +597,7 @@ var widget = (function () {
m += '[' + cv + s2ms(mp.au.currentTime) + ck + '/' + cv + s2ms(mp.au.duration) + ck + ']'; m += '[' + cv + s2ms(mp.au.currentTime) + ck + '/' + cv + s2ms(mp.au.duration) + ck + ']';
var o = mknod('input'); var o = document.createElement('input');
o.style.cssText = 'position:fixed;top:45%;left:48%;padding:1em;z-index:9'; o.style.cssText = 'position:fixed;top:45%;left:48%;padding:1em;z-index:9';
o.value = m; o.value = m;
document.body.appendChild(o); document.body.appendChild(o);
@@ -1151,7 +1135,7 @@ var audio_eq = (function () {
v = parseFloat(vs); v = parseFloat(vs);
if (isNaN(v) || v + '' != vs) if (isNaN(v) || v + '' != vs)
throw new Error('inval band'); throw 42;
if (isNaN(band)) if (isNaN(band))
r.amp = Math.round((v + step * 0.2) * 100) / 100; r.amp = Math.round((v + step * 0.2) * 100) / 100;
@@ -1353,7 +1337,7 @@ function play(tid, is_ev, seek, call_depth) {
return true; return true;
} }
catch (ex) { catch (ex) {
toast.err(0, 'playback failed: ' + ex); alert('playback failed: ' + ex);
} }
setclass(oid, 'play'); setclass(oid, 'play');
setTimeout(next_song, 500); setTimeout(next_song, 500);
@@ -1457,260 +1441,17 @@ function play_linked() {
}; };
(function () {
var d = mknod('div');
d.setAttribute('id', 'acc_info');
document.body.insertBefore(d, ebi('ops'));
})();
var fileman = (function () {
var bren = ebi('fren'),
bdel = ebi('fdel'),
bcut = ebi('fcut'),
bpst = ebi('fpst'),
r = {};
r.clip = null;
r.bus = new BroadcastChannel("fileman_bus");
r.render = function () {
if (r.clip === null)
r.clip = jread('fman_clip', []);
var sel = msel.getsel();
clmod(bren, 'en', sel.length == 1);
clmod(bdel, 'en', sel.length);
clmod(bcut, 'en', sel.length);
clmod(bpst, 'en', r.clip && r.clip.length);
bren.style.display = have_mv && has(perms, 'write') && has(perms, 'move') ? '' : 'none';
bdel.style.display = have_del && has(perms, 'delete') ? '' : 'none';
bcut.style.display = have_mv && has(perms, 'move') ? '' : 'none';
bpst.style.display = have_mv && has(perms, 'write') ? '' : 'none';
bpst.setAttribute('tt', 'paste ' + r.clip.length + ' items$NHotkey: ctrl-V');
ebi('wfm').style.display = QS('#wfm a.en:not([display])') ? '' : 'none';
};
r.rename = function (e) {
ev(e);
if (bren.style.display)
return toast.err(3, 'cannot rename:\nyou do not have “move” permission in this folder');
var sel = msel.getsel();
if (sel.length !== 1)
return toast.err(3, 'select exactly 1 item to rename');
var src = sel[0].vp;
if (src.endsWith('/'))
src = src.slice(0, -1);
var vsp = vsplit(src),
base = vsp[0],
ofn = vsp[1];
var fn = prompt('new filename:', ofn);
if (!fn || fn == ofn)
return toast.warn(1, 'rename aborted');
var dst = base + fn;
function rename_cb() {
if (this.readyState != XMLHttpRequest.DONE)
return;
if (this.status !== 200) {
var msg = this.responseText;
toast.err(9, 'rename failed:\n' + msg);
return;
}
toast.ok(2, 'rename OK');
treectl.goto(get_evpath());
}
var xhr = new XMLHttpRequest();
xhr.open('GET', src + '?move=' + dst, true);
xhr.onreadystatechange = rename_cb;
xhr.send();
};
r.delete = function (e) {
ev(e);
if (bdel.style.display)
return toast.err(3, 'cannot delete:\nyou do not have “delete” permission in this folder');
var sel = msel.getsel(),
vps = [];
for (var a = 0; a < sel.length; a++)
vps.push(sel[a].vp);
if (!sel.length)
return toast.err(3, 'select at least 1 item to delete');
if (!confirm('===== DANGER =====\nDELETE these ' + vps.length + ' items?\n\n' + vps.join('\n')))
return;
if (!confirm('Last chance! Delete?'))
return;
function deleter() {
var xhr = new XMLHttpRequest(),
vp = vps.shift();
if (!vp) {
toast.ok(2, 'delete OK');
treectl.goto(get_evpath());
return;
}
toast.inf(0, 'deleting ' + (vps.length + 1) + ' items\n\n' + vp);
xhr.open('GET', vp + '?delete', true);
xhr.onreadystatechange = delete_cb;
xhr.send();
}
function delete_cb() {
if (this.readyState != XMLHttpRequest.DONE)
return;
if (this.status !== 200) {
var msg = this.responseText;
toast.err(9, 'delete failed:\n' + msg);
return;
}
deleter();
}
deleter();
};
r.cut = function (e) {
ev(e);
if (bcut.style.display)
return toast.err(3, 'cannot cut:\nyou do not have “move” permission in this folder');
var sel = msel.getsel(),
vps = [];
if (!sel.length)
return toast.err(3, 'select at least 1 item to cut');
for (var a = 0; a < sel.length; a++) {
vps.push(sel[a].vp);
var cl = ebi(sel[a].id).closest('tr').classList,
inv = cl.contains('c1');
cl.remove(inv ? 'c1' : 'c2');
cl.add(inv ? 'c2' : 'c1');
}
toast.inf(1, 'cut ' + sel.length + ' items');
jwrite('fman_clip', vps);
r.tx(1);
};
r.paste = function (e) {
ev(e);
if (bpst.style.display)
return toast.err(3, 'cannot paste:\nyou do not have “write” permission in this folder');
if (!r.clip.length)
return toast.err(5, 'first cut some files/folders to paste\n\nnote: you can cut/paste across different browser tabs');
var req = [],
exists = [],
indir = [],
srcdir = vsplit(r.clip[0])[0],
links = QSA('#files tbody td:nth-child(2) a');
for (var a = 0, aa = links.length; a < aa; a++)
indir.push(links[a].getAttribute('name'));
for (var a = 0; a < r.clip.length; a++) {
var found = false;
for (var b = 0; b < indir.length; b++) {
if (r.clip[a].endsWith('/' + indir[b])) {
exists.push(r.clip[a]);
found = true;
}
}
if (!found)
req.push(r.clip[a]);
}
if (exists.length)
alert('these ' + exists.length + ' items cannot be pasted here (names already exist):\n\n' + exists.join('\n'));
if (!req.length)
return;
if (!confirm('paste these ' + req.length + ' items here?\n\n' + req.join('\n')))
return;
function paster() {
var xhr = new XMLHttpRequest(),
vp = req.shift();
if (!vp) {
toast.ok(2, 'paste OK');
treectl.goto(get_evpath());
r.tx(srcdir);
return;
}
toast.inf(0, 'pasting ' + (req.length + 1) + ' items\n\n' + vp);
var dst = get_evpath() + vp.split('/').slice(-1)[0];
xhr.open('GET', vp + '?move=' + dst, true);
xhr.onreadystatechange = paste_cb;
xhr.send();
}
function paste_cb() {
if (this.readyState != XMLHttpRequest.DONE)
return;
if (this.status !== 200) {
var msg = this.responseText;
toast.err(9, 'paste failed:\n' + msg);
return;
}
paster();
}
paster();
jwrite('fman_clip', []);
};
r.bus.onmessage = function (e) {
r.clip = null;
r.render();
var me = get_evpath();
if (e && e.data == me)
treectl.goto(e.data);
};
r.tx = function (msg) {
r.bus.postMessage(msg);
r.bus.onmessage();
};
bren.onclick = r.rename;
bdel.onclick = r.delete;
bcut.onclick = r.cut;
bpst.onclick = r.paste;
return r;
})();
var thegrid = (function () { var thegrid = (function () {
var lfiles = ebi('files'), var lfiles = ebi('files'),
gfiles = mknod('div'); gfiles = document.createElement('div');
gfiles.setAttribute('id', 'gfiles'); gfiles.setAttribute('id', 'gfiles');
gfiles.style.display = 'none'; gfiles.style.display = 'none';
gfiles.innerHTML = ( gfiles.innerHTML = (
'<div id="ghead">' + '<div id="ghead">' +
'<a href="#" class="tgl btn" id="gridsel" tt="enable file selection; ctrl-click a file to override$NHotkey: S">multiselect</a> &nbsp; zoom ' + '<a href="#" class="tgl btn" id="gridsel" tt="enable file selection; ctrl-click a file to override$NHotkey: S">multiselect</a> &nbsp; zoom ' +
'<a href="#" class="btn" z="-1.2" tt="Hotkey: shift-A">&ndash;</a> ' + '<a href="#" class="btn" z="-1.2" tt="Hotkey: A">&ndash;</a> ' +
'<a href="#" class="btn" z="1.2" tt="Hotkey: shift-D">+</a> &nbsp; sort by: ' + '<a href="#" class="btn" z="1.2" tt="Hotkey: D">+</a> &nbsp; sort by: ' +
'<a href="#" s="href">name</a>, ' + '<a href="#" s="href">name</a>, ' +
'<a href="#" s="sz">size</a>, ' + '<a href="#" s="sz">size</a>, ' +
'<a href="#" s="ts">date</a>, ' + '<a href="#" s="ts">date</a>, ' +
@@ -1736,7 +1477,7 @@ var thegrid = (function () {
r.setdirty(); r.setdirty();
}; };
ebi('griden').onclick = ebi('wtgrid').onclick = function (e) { ebi('griden').onclick = function (e) {
ev(e); ev(e);
r.en = !r.en; r.en = !r.en;
bcfg_set('griden', r.en); bcfg_set('griden', r.en);
@@ -1804,13 +1545,14 @@ var thegrid = (function () {
setsz(); setsz();
function gclick(e) { function gclick(e) {
if (ctrl(e)) if (e && (e.ctrlKey || e.metaKey))
return true; return true;
var oth = ebi(this.getAttribute('ref')), var oth = ebi(this.getAttribute('ref')),
href = this.getAttribute('href'), href = this.getAttribute('href'),
aplay = ebi('a' + oth.getAttribute('id')), aplay = ebi('a' + oth.getAttribute('id')),
is_img = /\.(gif|jpe?g|png|webp)(\?|$)/i.test(href), is_img = /\.(gif|jpe?g|png|webp)(\?|$)/i.test(href),
is_vid = /\.(av1|asf|avi|flv|m4v|mkv|mjpeg|mjpg|mpg|mpeg|mpg2|mpeg2|h264|avc|h265|hevc|mov|3gp|mp4|ts|mpegts|nut|ogv|ogm|rm|vob|webm|wmv)(\?|$)/i.test(href),
in_tree = null, in_tree = null,
have_sel = QS('#files tr.sel'), have_sel = QS('#files tr.sel'),
td = oth.closest('td').nextSibling, td = oth.closest('td').nextSibling,
@@ -1838,6 +1580,9 @@ var thegrid = (function () {
else if (in_tree && !have_sel) else if (in_tree && !have_sel)
in_tree.click(); in_tree.click();
else if (is_vid)
window.open(href + (href.indexOf('?') === -1 ? '?' : '&') + 'vcr', '_blank');
else if (!is_img && have_sel) else if (!is_img && have_sel)
window.open(href, '_blank'); window.open(href, '_blank');
@@ -1958,26 +1703,6 @@ var thegrid = (function () {
})(); })();
function tree_scrollto() {
var act = QS('#treeul a.hl'),
ul = act ? act.offsetParent : null;
if (!ul)
return;
var ctr = ebi('tree'),
em = parseFloat(getComputedStyle(act).fontSize),
top = act.offsetTop + ul.offsetTop,
min = top - 11 * em,
max = top - (ctr.offsetHeight - 10 * em);
if (ctr.scrollTop > min)
ctr.scrollTop = Math.floor(min);
else if (ctr.scrollTop < max)
ctr.scrollTop = Math.floor(max);
}
function tree_neigh(n) { function tree_neigh(n) {
var links = QSA('#treeul li>a+a'); var links = QSA('#treeul li>a+a');
if (!links.length) { if (!links.length) {
@@ -2003,7 +1728,6 @@ function tree_neigh(n) {
if (act >= links.length) if (act >= links.length)
act = 0; act = 0;
treectl.dir_cb = tree_scrollto;
links[act].click(); links[act].click();
} }
@@ -2023,66 +1747,13 @@ function tree_up() {
document.onkeydown = function (e) { document.onkeydown = function (e) {
var ae = document.activeElement, aet = ''; if (!document.activeElement || document.activeElement != document.body && document.activeElement.nodeName.toLowerCase() != 'a')
if (ae && ae != document.body)
aet = ae.nodeName.toLowerCase();
if (e.altKey || e.isComposing)
return; return;
if (QS('#bbox-overlay.visible')) if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
return; return;
var k = e.code + '', pos = -1, n; var k = (e.code + ''), pos = -1, n;
if (aet == 'tr' && ae.closest('#files')) {
var d = '';
if (k == 'ArrowUp') d = 'previous';
if (k == 'ArrowDown') d = 'next';
if (d) {
var el = ae[d + 'ElementSibling'];
if (el) {
el.focus();
if (ctrl(e))
document.documentElement.scrollTop += (d == 'next' ? 1 : -1) * el.offsetHeight;
if (e.shiftKey) {
clmod(el, 'sel', 't');
msel.selui();
}
return ev(e);
}
}
if (k == 'Space') {
clmod(ae, 'sel', 't');
msel.selui();
return ev(e);
}
if (k == 'KeyA' && ctrl(e)) {
var sel = msel.getsel(),
all = msel.getall();
msel.evsel(e, sel.length < all.length);
return ev(e);
}
}
if (aet && aet != 'a' && aet != 'tr')
return;
if (ctrl(e)) {
if (k == 'KeyX')
return fileman.cut();
if (k == 'KeyV')
return fileman.paste();
if (k == 'KeyK')
return fileman.delete();
return;
}
if (e.shiftKey && k != 'KeyA' && k != 'KeyD') if (e.shiftKey && k != 'KeyA' && k != 'KeyD')
return; return;
@@ -2122,9 +1793,6 @@ document.onkeydown = function (e) {
if (k == 'KeyT') if (k == 'KeyT')
return ebi('thumbs').click(); return ebi('thumbs').click();
if (k == 'F2')
return fileman.rename();
if (!treectl.hidden && (!e.shiftKey || !thegrid.en)) { if (!treectl.hidden && (!e.shiftKey || !thegrid.en)) {
if (k == 'KeyA') if (k == 'KeyA')
return QS('#twig').click(); return QS('#twig').click();
@@ -2261,7 +1929,7 @@ document.onkeydown = function (e) {
for (var b = 1; b < sconf[a].length; b++) { for (var b = 1; b < sconf[a].length; b++) {
var k = sconf[a][b][0], var k = sconf[a][b][0],
chk = 'srch_' + k + 'c', chk = 'srch_' + k + 'c',
tvs = ebi('srch_' + k + 'v').value.split(/ +/g); tvs = ebi('srch_' + k + 'v').value.split(/ /g);
if (!ebi(chk).checked) if (!ebi(chk).checked)
continue; continue;
@@ -2274,7 +1942,7 @@ document.onkeydown = function (e) {
q += ' and '; q += ' and ';
if (k == 'adv') { if (k == 'adv') {
q += tv.replace(/ +/g, " and ").replace(/([=!><]=?)/, " $1 "); q += tv.replace(/ /g, " and ").replace(/([=!><]=?)/, " $1 ");
continue; continue;
} }
@@ -2409,6 +2077,7 @@ document.onkeydown = function (e) {
ebi('files').innerHTML = orig_html; ebi('files').innerHTML = orig_html;
ebi('files').removeAttribute('q_raw'); ebi('files').removeAttribute('q_raw');
orig_html = null; orig_html = null;
msel.render();
reload_browser(); reload_browser();
} }
})(); })();
@@ -2554,7 +2223,7 @@ var treectl = (function () {
return; return;
if (this.status !== 200) { if (this.status !== 200) {
toast.err(0, "recvtree, http " + this.status + ": " + this.responseText); alert("http " + this.status + ": " + this.responseText);
return; return;
} }
@@ -2604,12 +2273,7 @@ var treectl = (function () {
var fun = treectl.dir_cb; var fun = treectl.dir_cb;
if (fun) { if (fun) {
treectl.dir_cb = null; treectl.dir_cb = null;
try { fun();
fun();
}
catch (ex) {
console.log("dir_cb failed", ex);
}
} }
} }
@@ -2630,7 +2294,7 @@ var treectl = (function () {
} }
function treego(e) { function treego(e) {
if (ctrl(e)) if (e && (e.ctrlKey || e.metaKey))
return true; return true;
ev(e); ev(e);
@@ -2676,7 +2340,7 @@ var treectl = (function () {
return; return;
if (this.status !== 200) { if (this.status !== 200) {
toast.err(0, "recvls, http " + this.status + ": " + this.responseText); alert("http " + this.status + ": " + this.responseText);
return; return;
} }
@@ -2731,7 +2395,6 @@ var treectl = (function () {
if (this.hpush) if (this.hpush)
hist_push(this.top); hist_push(this.top);
acct = res.acct;
apply_perms(res.perms); apply_perms(res.perms);
despin('#files'); despin('#files');
despin('#gfiles'); despin('#gfiles');
@@ -2743,6 +2406,7 @@ var treectl = (function () {
filecols.set_style(); filecols.set_style();
mukey.render(); mukey.render();
msel.render();
reload_tree(); reload_tree();
reload_browser(); reload_browser();
@@ -2847,26 +2511,9 @@ function despin(sel) {
function apply_perms(newperms) { function apply_perms(newperms) {
perms = newperms || []; perms = newperms || [];
var axs = [],
aclass = '>',
chk = ['read', 'write', 'rename', 'delete'];
for (var a = 0; a < chk.length; a++)
if (has(perms, chk[a]))
axs.push(chk[a].slice(0, 1).toUpperCase() + chk[a].slice(1));
axs = axs.join('-');
if (perms.length == 1) {
aclass = ' class="warn">';
axs += '-Only';
}
ebi('acc_info').innerHTML = '<span' + aclass + axs + ' access</span>' + (acct != '*' ?
'<a href="/?pw=x">Logout ' + acct + '</a>' : '<a href="/?h">Login</a>');
var o = QSA('#ops>a[data-perm], #u2footfoot'); var o = QSA('#ops>a[data-perm], #u2footfoot');
for (var a = 0; a < o.length; a++) { for (var a = 0; a < o.length; a++) {
var display = ''; var display = 'inline';
var needed = o[a].getAttribute('data-perm').split(' '); var needed = o[a].getAttribute('data-perm').split(' ');
for (var b = 0; b < needed.length; b++) { for (var b = 0; b < needed.length; b++) {
if (!has(perms, needed[b])) { if (!has(perms, needed[b])) {
@@ -2887,10 +2534,12 @@ function apply_perms(newperms) {
de = document.documentElement, de = document.documentElement,
tds = QSA('#u2conf td'); tds = QSA('#u2conf td');
/* good idea maybe
clmod(de, "read", have_read); clmod(de, "read", have_read);
clmod(de, "write", have_write); clmod(de, "write", have_write);
clmod(de, "nread", !have_read); clmod(de, "nread", !have_read);
clmod(de, "nwrite", !have_write); clmod(de, "nwrite", !have_write);
*/
for (var a = 0; a < tds.length; a++) { for (var a = 0; a < tds.length; a++) {
tds[a].style.display = tds[a].style.display =
@@ -2916,7 +2565,7 @@ function find_file_col(txt) {
for (var a = 0; a < tds.length; a++) { for (var a = 0; a < tds.length; a++) {
var spans = tds[a].getElementsByTagName('span'); var spans = tds[a].getElementsByTagName('span');
if (spans.length && spans[0].textContent == txt) { if (spans.length && spans[0].textContent == txt) {
min = (tds[a].getAttribute('class') || '').indexOf('min') !== -1; min = tds[a].getAttribute('class').indexOf('min') !== -1;
i = a; i = a;
break; break;
} }
@@ -2958,36 +2607,14 @@ function mk_files_header(taglist) {
var filecols = (function () { var filecols = (function () {
var hidden = jread('filecols', []), var hidden = jread('filecols', []);
tts = {
"c": "action buttons",
"dur": "duration",
"q": "quality / bitrate",
"Ac": "audio codec",
"Vc": "video codec",
"Res": "resolution",
"T": "filetype",
"aq": "audio quality / bitrate",
"vq": "video quality / bitrate",
"pixfmt": "subsampling / pixel structure",
"resw": "horizontal resolution",
"resh": "veritcal resolution",
"acs": "audio channels",
"hz": "sample rate"
};
var add_btns = function () { var add_btns = function () {
var ths = QSA('#files th>span'); var ths = QSA('#files th>span');
for (var a = 0, aa = ths.length; a < aa; a++) { for (var a = 0, aa = ths.length; a < aa; a++) {
var th = ths[a].parentElement, var th = ths[a].parentElement;
ttv = tts[ths[a].textContent];
th.innerHTML = '<div class="cfg"><a href="#">-</a></div>' + ths[a].outerHTML; th.innerHTML = '<div class="cfg"><a href="#">-</a></div>' + ths[a].outerHTML;
th.getElementsByTagName('a')[0].onclick = ev_row_tgl; th.getElementsByTagName('a')[0].onclick = ev_row_tgl;
if (ttv) {
th.setAttribute("tt", ttv);
th.setAttribute("ttd", "u");
}
} }
}; };
@@ -3010,10 +2637,7 @@ var filecols = (function () {
hcols = ebi('hcols'); hcols = ebi('hcols');
for (var a = 0; a < hidden.length; a++) { for (var a = 0; a < hidden.length; a++) {
var ttv = tts[hidden[a]], html.push('<a href="#" class="btn">' + esc(hidden[a]) + '</a>');
tta = ttv ? ' tt="' + ttv + '">' : '>';
html.push('<a href="#" class="btn"' + tta + esc(hidden[a]) + '</a>');
} }
hcols.previousSibling.style.display = html.length ? 'block' : 'none'; hcols.previousSibling.style.display = html.length ? 'block' : 'none';
hcols.innerHTML = html.join('\n'); hcols.innerHTML = html.join('\n');
@@ -3046,10 +2670,6 @@ var filecols = (function () {
for (var b = 0, bb = tds.length; b < bb; b++) for (var b = 0, bb = tds.length; b < bb; b++)
tds[b].setAttribute('class', cls); tds[b].setAttribute('class', cls);
} }
if (window['tt']) {
tt.att(ebi('hcols'));
tt.att(QS('#files thead'));
}
}; };
set_style(); set_style();
@@ -3240,7 +2860,7 @@ var arcfmt = (function () {
["tar", "tar", "plain gnutar file"], ["tar", "tar", "plain gnutar file"],
["zip", "zip=utf8", "zip with utf8 filenames (maybe wonky on windows 7 and older)"], ["zip", "zip=utf8", "zip with utf8 filenames (maybe wonky on windows 7 and older)"],
["zip_dos", "zip", "zip with traditional cp437 filenames, for really old software"], ["zip_dos", "zip", "zip with traditional cp437 filenames, for really old software"],
["zip_crc", "zip=crc", "cp437 with crc32 computed early,$Nfor MS-DOS PKZIP v2.04g (october 1993)$N(takes longer to process before download can start)"] ["zip_crc", "zip=crc", "cp437 with crc32 computed early for truly ancient software$N(takes longer to process before download can start)"]
]; ];
for (var a = 0; a < fmts.length; a++) { for (var a = 0; a < fmts.length; a++) {
@@ -3272,7 +2892,7 @@ var arcfmt = (function () {
var ofs = href.lastIndexOf('?'); var ofs = href.lastIndexOf('?');
if (ofs < 0) if (ofs < 0)
throw new Error('missing arg in url'); throw 'missing arg in url';
o.setAttribute("href", href.slice(0, ofs + 1) + arg); o.setAttribute("href", href.slice(0, ofs + 1) + arg);
o.textContent = fmt.split('_')[0]; o.textContent = fmt.split('_')[0];
@@ -3309,73 +2929,41 @@ var arcfmt = (function () {
var msel = (function () { var msel = (function () {
var r = {}; function getsel() {
r.sel = null; var names = [],
r.all = null; links = QSA('#files tbody tr.sel td:nth-child(2) a');
r.load = function () { for (var a = 0, aa = links.length; a < aa; a++)
if (r.sel) names.push(links[a].getAttribute('href').replace(/\/$/, "").split('/').slice(-1));
return;
r.sel = []; return names;
r.all = [];
var links = QSA('#files tbody td:nth-child(2) a:last-child'),
vbase = get_evpath();
for (var a = 0, aa = links.length; a < aa; a++) {
var href = links[a].getAttribute('href').replace(/\/$/, ""),
item = {};
item.id = links[a].getAttribute('id');
item.sel = links[a].closest('tr').classList.contains('sel');
item.vp = href.indexOf('/') !== -1 ? href : vbase + href;
item.name = href.split('/').slice(-1);
r.all.push(item);
if (item.sel)
r.sel.push(item);
links[a].setAttribute('name', item.name);
links[a].closest('tr').setAttribute('tabindex', '0');
}
};
r.getsel = function () {
r.load();
return r.sel;
};
r.getall = function () {
r.load();
return r.all;
};
r.selui = function () {
r.sel = r.all = null;
clmod(ebi('wtoggle'), 'sel', r.getsel().length);
thegrid.loadsel();
fileman.render();
} }
r.seltgl = function (e) { function selui() {
clmod(ebi('wtoggle'), 'sel', getsel().length);
thegrid.loadsel();
}
function seltgl(e) {
ev(e); ev(e);
var tr = this.parentNode; var tr = this.parentNode;
clmod(tr, 'sel', 't'); clmod(tr, 'sel', 't');
r.selui(); selui();
} }
r.evsel = function (e, fun) { function evsel(e, fun) {
ev(e); ev(e);
var trs = QSA('#files tbody tr'); var trs = QSA('#files tbody tr');
for (var a = 0, aa = trs.length; a < aa; a++) for (var a = 0, aa = trs.length; a < aa; a++)
clmod(trs[a], 'sel', fun); clmod(trs[a], 'sel', fun);
r.selui(); selui();
} }
ebi('selall').onclick = function (e) { ebi('selall').onclick = function (e) {
r.evsel(e, "add"); evsel(e, "add");
}; };
ebi('selinv').onclick = function (e) { ebi('selinv').onclick = function (e) {
r.evsel(e, "t"); evsel(e, "t");
}; };
ebi('selzip').onclick = function (e) { ebi('selzip').onclick = function (e) {
ev(e); ev(e);
var names = r.getsel(), var names = getsel(),
arg = ebi('selzip').getAttribute('fmt'), arg = ebi('selzip').getAttribute('fmt'),
txt = names.join('\n'), txt = names.join('\n'),
frm = mknod('form'); frm = mknod('form');
@@ -3398,17 +2986,16 @@ var msel = (function () {
console.log(txt); console.log(txt);
frm.submit(); frm.submit();
}; };
r.render = function () { function render() {
var tds = QSA('#files tbody td+td+td'); var tds = QSA('#files tbody td+td+td');
for (var a = 0, aa = tds.length; a < aa; a++) { for (var a = 0, aa = tds.length; a < aa; a++) {
tds[a].onclick = r.seltgl; tds[a].onclick = seltgl;
} }
r.selui();
arcfmt.render(); arcfmt.render();
fileman.render();
ebi('selzip').style.display = ebi('unsearch') ? 'none' : '';
} }
return r; return {
"render": render
};
})(); })();
@@ -3467,7 +3054,7 @@ function reload_browser(not_mp) {
var oo = QSA('#files>tbody>tr>td:nth-child(3)'); var oo = QSA('#files>tbody>tr>td:nth-child(3)');
for (var a = 0, aa = oo.length; a < aa; a++) { for (var a = 0, aa = oo.length; a < aa; a++) {
var sz = oo[a].textContent.replace(/ +/g, ""), var sz = oo[a].textContent.replace(/ /g, ""),
hsz = sz.replace(/\B(?=(\d{3})+(?!\d))/g, " "); hsz = sz.replace(/\B(?=(\d{3})+(?!\d))/g, " ");
oo[a].textContent = hsz; oo[a].textContent = hsz;
@@ -3483,8 +3070,8 @@ function reload_browser(not_mp) {
up2k.set_fsearch(); up2k.set_fsearch();
thegrid.setdirty(); thegrid.setdirty();
msel.render();
} }
reload_browser(true); reload_browser(true);
mukey.render(); mukey.render();
msel.render();
play_linked(); play_linked();

View File

@@ -8,137 +8,6 @@ html, body {
font-family: sans-serif; font-family: sans-serif;
line-height: 1.5em; line-height: 1.5em;
} }
#tt, #toast {
position: fixed;
max-width: 34em;
background: #222;
border: 0 solid #777;
box-shadow: 0 .2em .5em #222;
border-radius: .4em;
z-index: 9001;
}
#tt {
overflow: hidden;
margin-top: 1em;
padding: 0 1.3em;
height: 0;
opacity: .1;
transition: opacity 0.14s, height 0.14s, padding 0.14s;
}
#toast {
top: 1.4em;
right: -1em;
line-height: 1.5em;
padding: 1em 1.3em;
border-width: .4em 0;
transform: translateX(100%);
transition:
transform .4s cubic-bezier(.2, 1.2, .5, 1),
right .4s cubic-bezier(.2, 1.2, .5, 1);
text-shadow: 1px 1px 0 #000;
color: #fff;
}
#toastc {
display: inline-block;
position: absolute;
overflow: hidden;
left: 0;
width: 0;
opacity: 0;
padding: .3em 0;
margin: -.3em 0 0 0;
line-height: 1.5em;
color: #000;
border: none;
outline: none;
text-shadow: none;
border-radius: .5em 0 0 .5em;
transition: left .3s, width .3s, padding .3s, opacity .3s;
}
#toast.vis {
right: 1.3em;
transform: unset;
}
#toast.vis #toastc {
left: -2em;
width: .4em;
padding: .3em .8em;
opacity: 1;
}
#toast.inf {
background: #07a;
border-color: #0be;
}
#toast.inf #toastc {
background: #0be;
}
#toast.ok {
background: #4a0;
border-color: #8e4;
}
#toast.ok #toastc {
background: #8e4;
}
#toast.warn {
background: #970;
border-color: #fc0;
}
#toast.warn #toastc {
background: #fc0;
}
#toast.err {
background: #900;
border-color: #d06;
}
#toast.err #toastc {
background: #d06;
}
#tt.b {
padding: 0 2em;
border-radius: .5em;
box-shadow: 0 .2em 1em #000;
}
#tt.show {
padding: 1em 1.3em;
border-width: .4em 0;
height: auto;
opacity: 1;
}
#tt.show.b {
padding: 1.5em 2em;
border-width: .5em 0;
}
#tt code {
background: #3c3c3c;
padding: .1em .3em;
border-top: 1px solid #777;
border-radius: .3em;
line-height: 1.7em;
}
#tt em {
color: #f6a;
}
html.light #tt {
background: #fff;
border-color: #888 #000 #777 #000;
}
html.light #tt,
html.light #toast {
box-shadow: 0 .3em 1em rgba(0,0,0,0.4);
}
html.light #tt code {
background: #060;
color: #fff;
}
html.light #tt em {
color: #d38;
}
#mtw { #mtw {
display: none; display: none;
} }
@@ -157,7 +26,7 @@ pre, code, a {
code { code {
font-size: .96em; font-size: .96em;
} }
pre, code, tt { pre, code {
font-family: 'scp', monospace, monospace; font-family: 'scp', monospace, monospace;
white-space: pre-wrap; white-space: pre-wrap;
word-break: break-all; word-break: break-all;
@@ -297,7 +166,7 @@ small {
z-index: 99; z-index: 99;
position: relative; position: relative;
display: inline-block; display: inline-block;
font-family: 'scp', monospace, monospace; font-family: monospace, monospace;
font-weight: bold; font-weight: bold;
font-size: 1.3em; font-size: 1.3em;
line-height: .1em; line-height: .1em;

View File

@@ -14,9 +14,9 @@
<a id="lightswitch" href="#">go dark</a> <a id="lightswitch" href="#">go dark</a>
<a id="navtoggle" href="#">hide nav</a> <a id="navtoggle" href="#">hide nav</a>
{%- if edit %} {%- if edit %}
<a id="save" href="?edit" tt="Hotkey: ctrl-s">save</a> <a id="save" href="?edit">save</a>
<a id="sbs" href="#" tt="editor and preview side by side">sbs</a> <a id="sbs" href="#">sbs</a>
<a id="nsbs" href="#" tt="switch between editor and preview$NHotkey: ctrl-e">editor</a> <a id="nsbs" href="#">editor</a>
<div id="toolsbox"> <div id="toolsbox">
<a id="tools" href="#">tools</a> <a id="tools" href="#">tools</a>
<a id="fmt_table" href="#">prettify table (ctrl-k)</a> <a id="fmt_table" href="#">prettify table (ctrl-k)</a>
@@ -26,8 +26,8 @@
<a id="help" href="#">help</a> <a id="help" href="#">help</a>
</div> </div>
{%- else %} {%- else %}
<a href="?edit" tt="good: higher performance$Ngood: same document width as viewer$Nbad: assumes you know markdown">edit (basic)</a> <a href="?edit">edit (basic)</a>
<a href="?edit2" tt="not in-house so probably less buggy">edit (fancy)</a> <a href="?edit2">edit (fancy)</a>
<a href="?raw">view raw</a> <a href="?raw">view raw</a>
{%- endif %} {%- endif %}
</div> </div>
@@ -131,18 +131,18 @@ var md_opt = {
}; };
(function () { (function () {
var l = localStorage, var btn = document.getElementById("lightswitch");
drk = l.getItem('lightmode') != 1, var toggle = function (e) {
btn = document.getElementById("lightswitch"), if (e) e.preventDefault();
f = function (e) { var dark = !document.documentElement.getAttribute("class");
if (e) { e.preventDefault(); drk = !drk; } document.documentElement.setAttribute("class", dark ? "dark" : "");
document.documentElement.setAttribute("class", drk? "dark":"light"); btn.innerHTML = "go " + (dark ? "light" : "dark");
btn.innerHTML = "go " + (drk ? "light":"dark"); if (window.localStorage)
l.setItem('lightmode', drk? 0:1); localStorage.setItem('lightmode', dark ? 0 : 1);
}; };
btn.onclick = toggle;
btn.onclick = f; if (window.localStorage && localStorage.getItem('lightmode') != 1)
f(); toggle();
})(); })();
</script> </script>

View File

@@ -176,7 +176,7 @@ function md_plug_err(ex, js) {
var lns = js.split('\n'); var lns = js.split('\n');
if (ln < lns.length) { if (ln < lns.length) {
o = mknod('span'); o = mknod('span');
o.style.cssText = "color:#ac2;font-size:.9em;font-family:'scp',monospace,monospace;display:block"; o.style.cssText = 'color:#ac2;font-size:.9em;font-family:scp;display:block';
o.textContent = lns[ln - 1]; o.textContent = lns[ln - 1];
} }
} }
@@ -530,6 +530,3 @@ dom_navtgl.onclick = function () {
if (sread('hidenav') == 1) if (sread('hidenav') == 1)
dom_navtgl.onclick(); dom_navtgl.onclick();
if (window['tt'])
tt.init();

View File

@@ -84,10 +84,13 @@ html.dark #save.force-save {
#save.disabled { #save.disabled {
opacity: .4; opacity: .4;
} }
#helpbox { #helpbox,
#toast {
background: #f7f7f7; background: #f7f7f7;
border-radius: .4em; border-radius: .4em;
z-index: 9001; z-index: 9001;
}
#helpbox {
display: none; display: none;
position: fixed; position: fixed;
padding: 2em; padding: 2em;
@@ -104,7 +107,19 @@ html.dark #save.force-save {
} }
html.dark #helpbox { html.dark #helpbox {
box-shadow: 0 .5em 2em #444; box-shadow: 0 .5em 2em #444;
}
html.dark #helpbox,
html.dark #toast {
background: #222; background: #222;
border: 1px solid #079; border: 1px solid #079;
border-width: 1px 0; border-width: 1px 0;
} }
#toast {
font-weight: bold;
text-align: center;
padding: .6em 0;
position: fixed;
top: 30%;
transition: opacity 0.2s ease-in-out;
opacity: 1;
}

View File

@@ -236,7 +236,7 @@ function Modpoll() {
var skip = null; var skip = null;
if (toast.visible) if (ebi('toast'))
skip = 'toast'; skip = 'toast';
else if (this.skip_one) else if (this.skip_one)
@@ -291,9 +291,10 @@ function Modpoll() {
"Press F5 or CTRL-R to refresh the page,<br />" + "Press F5 or CTRL-R to refresh the page,<br />" +
"replacing your document with the server copy.", "replacing your document with the server copy.",
"You can close this message to ignore and contnue." "You can click this message to ignore and contnue."
]; ];
return toast.warn(0, "<p>" + msg.join('</p>\n<p>') + '</p>'); return toast(false, "box-shadow:0 1em 2em rgba(64,64,64,0.8);font-weight:normal",
36, "<p>" + msg.join('</p>\n<p>') + '</p>');
} }
console.log('modpoll eq'); console.log('modpoll eq');
@@ -322,12 +323,16 @@ function save(e) {
var save_btn = ebi("save"), var save_btn = ebi("save"),
save_cls = save_btn.getAttribute('class') + ''; save_cls = save_btn.getAttribute('class') + '';
if (save_cls.indexOf('disabled') >= 0) if (save_cls.indexOf('disabled') >= 0) {
return toast.inf(2, "no changes"); toast(true, ";font-size:2em;color:#c90", 9, "no changes");
return;
}
var force = (save_cls.indexOf('force-save') >= 0); var force = (save_cls.indexOf('force-save') >= 0);
if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) {
return toast.inf(3, 'aborted'); alert('ok, aborted');
return;
}
var txt = dom_src.value; var txt = dom_src.value;
@@ -352,15 +357,18 @@ function save_cb() {
if (this.readyState != XMLHttpRequest.DONE) if (this.readyState != XMLHttpRequest.DONE)
return; return;
if (this.status !== 200) if (this.status !== 200) {
return alert('Error! The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, "")); alert('Error! The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
return;
}
var r; var r;
try { try {
r = JSON.parse(this.responseText); r = JSON.parse(this.responseText);
} }
catch (ex) { catch (ex) {
return alert('Failed to parse reply from server:\n\n' + this.responseText); alert('Failed to parse reply from server:\n\n' + this.responseText);
return;
} }
if (!r.ok) { if (!r.ok) {
@@ -435,10 +443,46 @@ function savechk_cb() {
last_modified = this.lastmod; last_modified = this.lastmod;
server_md = this.txt; server_md = this.txt;
draw_md(); draw_md();
toast.ok(2, 'save OK' + (this.ntry ? '\nattempt ' + this.ntry : '')); toast(true, ";font-size:6em;font-family:serif;color:#9b4", 4,
'OK✔<span style="font-size:.2em;color:#999;position:absolute">' + this.ntry + '</span>');
modpoll.disabled = false; modpoll.disabled = false;
} }
function toast(autoclose, style, width, msg) {
var ok = ebi("toast");
if (ok)
ok.parentNode.removeChild(ok);
style = "width:" + width + "em;left:calc(50% - " + (width / 2) + "em);" + style;
ok = mknod('div');
ok.setAttribute('id', 'toast');
ok.setAttribute('style', style);
ok.innerHTML = msg;
var parent = ebi('m');
document.documentElement.appendChild(ok);
var hide = function (delay) {
delay = delay || 0;
setTimeout(function () {
ok.style.opacity = 0;
}, delay);
setTimeout(function () {
if (ok.parentNode)
ok.parentNode.removeChild(ok);
}, delay + 250);
}
ok.onclick = function () {
hide(0);
};
if (autoclose)
hide(500);
}
// firefox bug: initial selection offset isn't cleared properly through js // firefox bug: initial selection offset isn't cleared properly through js
var ff_clearsel = (function () { var ff_clearsel = (function () {
@@ -717,7 +761,7 @@ function fmt_table(e) {
var ind2 = tab[a].match(re_ind)[0]; var ind2 = tab[a].match(re_ind)[0];
if (ind != ind2 && a != 1) // the table can be a list entry or something, ignore [0] if (ind != ind2 && a != 1) // the table can be a list entry or something, ignore [0]
return toast.err(7, err + 'indentation mismatch on row#2 and ' + row_name + ',\n' + tab[a]); return alert(err + 'indentation mismatch on row#2 and ' + row_name + ',\n' + tab[a]);
var t = tab[a].slice(ind.length); var t = tab[a].slice(ind.length);
t = t.replace(re_lpipe, ""); t = t.replace(re_lpipe, "");
@@ -727,7 +771,7 @@ function fmt_table(e) {
if (a == 0) if (a == 0)
ncols = tab[a].length; ncols = tab[a].length;
else if (ncols < tab[a].length) else if (ncols < tab[a].length)
return toast.err(7, err + 'num.columns(' + row_name + ') exceeding row#2; ' + ncols + ' < ' + tab[a].length); return alert(err + 'num.columns(' + row_name + ') exceeding row#2; ' + ncols + ' < ' + tab[a].length);
// if row has less columns than row2, fill them in // if row has less columns than row2, fill them in
while (tab[a].length < ncols) while (tab[a].length < ncols)
@@ -744,7 +788,7 @@ function fmt_table(e) {
for (var col = 0; col < tab[1].length; col++) { for (var col = 0; col < tab[1].length; col++) {
var m = tab[1][col].match(re_align); var m = tab[1][col].match(re_align);
if (!m) if (!m)
return toast.err(7, err + 'invalid column specification, row#2, col ' + (col + 1) + ', [' + tab[1][col] + ']'); return alert(err + 'invalid column specification, row#2, col ' + (col + 1) + ', [' + tab[1][col] + ']');
if (m[2]) { if (m[2]) {
if (m[1]) if (m[1])
@@ -832,9 +876,10 @@ function mark_uni(e) {
ptn = new RegExp('([^' + js_uni_whitelist + ']+)', 'g'), ptn = new RegExp('([^' + js_uni_whitelist + ']+)', 'g'),
mod = txt.replace(/\r/g, "").replace(ptn, "\u2588\u2770$1\u2771"); mod = txt.replace(/\r/g, "").replace(ptn, "\u2588\u2770$1\u2771");
if (txt == mod) if (txt == mod) {
return toast.inf(5, 'no results; no modifications were made'); alert('no results; no modifications were made');
return;
}
dom_src.value = mod; dom_src.value = mod;
} }
@@ -848,9 +893,10 @@ function iter_uni(e) {
re = new RegExp('([^' + js_uni_whitelist + ']+)'), re = new RegExp('([^' + js_uni_whitelist + ']+)'),
m = re.exec(txt.slice(ofs)); m = re.exec(txt.slice(ofs));
if (!m) if (!m) {
return toast.inf(5, 'no more hits from cursor onwards'); alert('no more hits from cursor onwards');
return;
}
ofs += m.index; ofs += m.index;
dom_src.setSelectionRange(ofs, ofs + m[0].length, "forward"); dom_src.setSelectionRange(ofs, ofs + m[0].length, "forward");
@@ -878,9 +924,10 @@ function cfg_uni(e) {
(function () { (function () {
function keydown(ev) { function keydown(ev) {
ev = ev || window.event; ev = ev || window.event;
var kc = ev.code || ev.keyCode || ev.which; var kc = ev.keyCode || ev.which;
//console.log(ev.key, ev.code, ev.keyCode, ev.which); var ctrl = ev.ctrlKey || ev.metaKey;
if (ctrl(ev) && (ev.code == "KeyS" || kc == 83)) { //console.log(ev.code, kc);
if (ctrl && (ev.code == "KeyS" || kc == 83)) {
save(); save();
return false; return false;
} }
@@ -889,15 +936,23 @@ function cfg_uni(e) {
if (d) if (d)
d.click(); d.click();
} }
if (document.activeElement != dom_src) if (document.activeElement == dom_src) {
return true; if (ev.code == "Tab" || kc == 9) {
md_indent(ev.shiftKey);
if (ctrl(ev)) { return false;
if (ev.code == "KeyH" || kc == 72) { }
if (ctrl && (ev.code == "KeyH" || kc == 72)) {
md_header(ev.shiftKey); md_header(ev.shiftKey);
return false; return false;
} }
if (ev.code == "KeyZ" || kc == 90) { if (!ctrl && (ev.code == "Home" || kc == 36)) {
md_home(ev.shiftKey);
return false;
}
if (!ctrl && !ev.shiftKey && (ev.code == "Enter" || kc == 13)) {
return md_newline();
}
if (ctrl && (ev.code == "KeyZ" || kc == 90)) {
if (ev.shiftKey) if (ev.shiftKey)
action_stack.redo(); action_stack.redo();
else else
@@ -905,45 +960,33 @@ function cfg_uni(e) {
return false; return false;
} }
if (ev.code == "KeyY" || kc == 89) { if (ctrl && (ev.code == "KeyY" || kc == 89)) {
action_stack.redo(); action_stack.redo();
return false; return false;
} }
if (ev.code == "KeyK") { if (!ctrl && !ev.shiftKey && kc == 8) {
return md_backspace();
}
if (ctrl && (ev.code == "KeyK")) {
fmt_table(); fmt_table();
return false; return false;
} }
if (ev.code == "KeyU") { if (ctrl && (ev.code == "KeyU")) {
iter_uni(); iter_uni();
return false; return false;
} }
if (ev.code == "KeyE") { if (ctrl && (ev.code == "KeyE")) {
dom_nsbs.click(); dom_nsbs.click();
//fmt_table();
return false; return false;
} }
var up = ev.code == "ArrowUp" || kc == 38; var up = ev.code == "ArrowUp" || kc == 38;
var dn = ev.code == "ArrowDown" || kc == 40; var dn = ev.code == "ArrowDown" || kc == 40;
if (up || dn) { if (ctrl && (up || dn)) {
md_p_jump(dn); md_p_jump(dn);
return false; return false;
} }
} }
else {
if (ev.code == "Tab" || kc == 9) {
md_indent(ev.shiftKey);
return false;
}
if (ev.code == "Home" || kc == 36) {
md_home(ev.shiftKey);
return false;
}
if (!ev.shiftKey && (ev.code == "Enter" || kc == 13)) {
return md_newline();
}
if (!ev.shiftKey && kc == 8) {
return md_backspace();
}
}
} }
document.onkeydown = keydown; document.onkeydown = keydown;
ebi('save').onclick = save; ebi('save').onclick = save;

View File

@@ -30,15 +30,16 @@ var md_opt = {
}; };
var lightswitch = (function () { var lightswitch = (function () {
var l = localStorage, var fun = function () {
drk = l.getItem('lightmode') != 1, var dark = !document.documentElement.getAttribute("class");
f = function (e) { document.documentElement.setAttribute("class", dark ? "dark" : "");
if (e) drk = !drk; if (window.localStorage)
document.documentElement.setAttribute("class", drk? "dark":"light"); localStorage.setItem('lightmode', dark ? 0 : 1);
l.setItem('lightmode', drk? 0:1); };
}; if (window.localStorage && localStorage.getItem('lightmode') != 1)
f(); fun();
return f;
return fun;
})(); })();
</script> </script>

View File

@@ -106,12 +106,15 @@ function md_changed(mde, on_srv) {
function save(mde) { function save(mde) {
var save_btn = QS('.editor-toolbar button.save'); var save_btn = QS('.editor-toolbar button.save');
if (save_btn.classList.contains('disabled')) if (save_btn.classList.contains('disabled')) {
return toast.inf(2, 'no changes'); alert('there is nothing to save');
return;
}
var force = save_btn.classList.contains('force-save'); var force = save_btn.classList.contains('force-save');
if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) if (force && !confirm('confirm that you wish to lose the changes made on the server since you opened this document')) {
return toast.inf(3, 'aborted'); alert('ok, aborted');
return;
}
var txt = mde.value(); var txt = mde.value();
@@ -135,15 +138,18 @@ function save_cb() {
if (this.readyState != XMLHttpRequest.DONE) if (this.readyState != XMLHttpRequest.DONE)
return; return;
if (this.status !== 200) if (this.status !== 200) {
return alert('Error! The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, "")); alert('Error! The file was NOT saved.\n\n' + this.status + ": " + (this.responseText + '').replace(/^<pre>/, ""));
return;
}
var r; var r;
try { try {
r = JSON.parse(this.responseText); r = JSON.parse(this.responseText);
} }
catch (ex) { catch (ex) {
return alert('Failed to parse reply from server:\n\n' + this.responseText); alert('Failed to parse reply from server:\n\n' + this.responseText);
return;
} }
if (!r.ok) { if (!r.ok) {

View File

@@ -35,7 +35,7 @@
</table> </table>
</td></tr></table> </td></tr></table>
<div class="btns"> <div class="btns">
<a href="/?stack">dump stack</a> <a href="{{ avol[0] }}?stack">dump stack</a>
</div> </div>
{%- endif %} {%- endif %}
@@ -68,7 +68,7 @@
</div> </div>
<script> <script>
if (localStorage.getItem('lightmode') != 1) if (window.localStorage && localStorage.getItem('lightmode') != 1)
document.documentElement.setAttribute("class", "dark"); document.documentElement.setAttribute("class", "dark");
</script> </script>

View File

@@ -225,7 +225,7 @@ function U2pvis(act, btns) {
this.hashed = function (fobj) { this.hashed = function (fobj) {
var fo = this.tab[fobj.n], var fo = this.tab[fobj.n],
nb = fo.bt * (++fo.nh / fo.cb.length), nb = fo.bt * (++fo.nh / fo.cb.length),
p = this.perc(nb, 0, fobj.size, fobj.t_hashing); p = this.perc(nb, 0, fobj.size, fobj.t1);
fo.hp = '{0}%, {1}, {2} MB/s'.format( fo.hp = '{0}%, {1}, {2} MB/s'.format(
p[0].toFixed(2), p[1], p[2].toFixed(2) p[0].toFixed(2), p[1], p[2].toFixed(2)
@@ -248,7 +248,7 @@ function U2pvis(act, btns) {
fo.cb[nchunk] = cbd; fo.cb[nchunk] = cbd;
fo.bd += delta; fo.bd += delta;
var p = this.perc(fo.bd, fo.bd0, fo.bt, fobj.t_uploading); var p = this.perc(fo.bd, fo.bd0, fo.bt, fobj.t3);
fo.hp = '{0}%, {1}, {2} MB/s'.format( fo.hp = '{0}%, {1}, {2} MB/s'.format(
p[0].toFixed(2), p[1], p[2].toFixed(2) p[0].toFixed(2), p[1], p[2].toFixed(2)
); );
@@ -290,7 +290,8 @@ function U2pvis(act, btns) {
if (this.is_act(this.tab[a].in)) if (this.is_act(this.tab[a].in))
console.log("tab %d/%d = sz %s", a, aa, this.tab[a].bt); console.log("tab %d/%d = sz %s", a, aa, this.tab[a].bt);
throw new Error('see console'); console.log("a");
throw 42;
} }
obj.innerHTML = fo.hp; obj.innerHTML = fo.hp;
@@ -303,8 +304,9 @@ function U2pvis(act, btns) {
oldcat = fo.in, oldcat = fo.in,
bz_act = this.act == "bz"; bz_act = this.act == "bz";
if (oldcat == newcat) if (oldcat == newcat) {
return; throw 42;
}
fo.in = newcat; fo.in = newcat;
this.ctr[oldcat]--; this.ctr[oldcat]--;
@@ -317,7 +319,7 @@ function U2pvis(act, btns) {
this.addrow(nfile); this.addrow(nfile);
} }
else if (this.is_act(oldcat)) { else if (this.is_act(oldcat)) {
while (this.head < Math.min(this.tab.length, this.tail) && this.precard[this.tab[this.head].in]) while (this.head < Math.min(this.tab.length, this.tail) && (this.head == nfile || !this.is_act(this.tab[this.head].in)))
this.head++; this.head++;
if (!bz_act) { if (!bz_act) {
@@ -325,10 +327,9 @@ function U2pvis(act, btns) {
tr.parentNode.removeChild(tr); tr.parentNode.removeChild(tr);
} }
} }
else return; if (bz_act) {
if (bz_act)
this.bzw(); this.bzw();
}
}; };
this.bzw = function () { this.bzw = function () {
@@ -342,8 +343,7 @@ function U2pvis(act, btns) {
while (this.head - first > this.wsz) { while (this.head - first > this.wsz) {
var obj = ebi('f' + (first++)); var obj = ebi('f' + (first++));
if (obj) obj.parentNode.removeChild(obj);
obj.parentNode.removeChild(obj);
} }
while (last - this.tail < this.wsz && last < this.tab.length - 2) { while (last - this.tail < this.wsz && last < this.tab.length - 2) {
var obj = ebi('f' + (++last)); var obj = ebi('f' + (++last));
@@ -376,8 +376,6 @@ function U2pvis(act, btns) {
this.changecard = function (card) { this.changecard = function (card) {
this.act = card; this.act = card;
this.precard = has(["ok", "ng", "done"], this.act) ? {} : this.act == "bz" ? { "ok": 1, "ng": 1 } : { "ok": 1, "ng": 1, "bz": 1 };
this.postcard = has(["ok", "ng", "done"], this.act) ? { "bz": 1, "q": 1 } : this.act == "bz" ? { "q": 1 } : {};
this.head = -1; this.head = -1;
this.tail = -1; this.tail = -1;
var html = []; var html = [];
@@ -392,23 +390,22 @@ function U2pvis(act, btns) {
} }
} }
if (this.head == -1) { if (this.head == -1) {
var precard = has(["ok", "ng", "done"], this.act) ? {} : this.act == "bz" ? { "ok": 1, "ng": 1 } : { "ok": 1, "ng": 1, "bz": 1 },
postcard = has(["ok", "ng", "done"], this.act) ? { "bz": 1, "q": 1 } : this.act == "bz" ? { "q": 1 } : {};
for (var a = 0; a < this.tab.length; a++) { for (var a = 0; a < this.tab.length; a++) {
var rt = this.tab[a].in; var rt = this.tab[a].in;
if (this.precard[rt]) { if (precard[rt]) {
this.head = a + 1; this.head = a + 1;
this.tail = a; this.tail = a;
} }
else if (this.postcard[rt]) { else if (postcard[rt]) {
this.head = a; this.head = a;
this.tail = a - 1; this.tail = a - 1;
break; break;
} }
} }
} }
if (this.head < 0)
this.head = 0;
if (card == "bz") { if (card == "bz") {
for (var a = this.head - 1; a >= this.head - this.wsz && a >= 0; a--) { for (var a = this.head - 1; a >= this.head - this.wsz && a >= 0; a--) {
html.unshift(this.genrow(a, true).replace(/><td>/, "><td>a ")); html.unshift(this.genrow(a, true).replace(/><td>/, "><td>a "));
@@ -455,17 +452,6 @@ function U2pvis(act, btns) {
that.changecard(newtab); that.changecard(newtab);
}; };
} }
this.changecard(this.act);
}
function fsearch_explain(e) {
ev(e);
if (!has(perms, 'write'))
return alert('your access to this folder is Read-Only\n\n' + (acct == '*' ? 'you are currently not logged in' : 'you are currently logged in as ' + acct));
alert('you are currently in file-search mode\n\nswitch to upload-mode by clicking the green magnifying glass (next to the big yellow search button), and then refresh\n\nsorry');
} }
@@ -496,9 +482,8 @@ function up2k_init(subtle) {
shame = 'your browser is impressively ancient'; shame = 'your browser is impressively ancient';
// upload ui hidden by default, clicking the header shows it // upload ui hidden by default, clicking the header shows it
var got_deps = false;
function init_deps() { function init_deps() {
if (!got_deps && !subtle && !window.asmCrypto) { if (!subtle && !window.asmCrypto) {
var fn = 'sha512.' + sha_js + '.js'; var fn = 'sha512.' + sha_js + '.js';
showmodal('<h1>loading ' + fn + '</h1><h2>since ' + shame + '</h2><h4>thanks chrome</h4>'); showmodal('<h1>loading ' + fn + '</h1><h2>since ' + shame + '</h2><h4>thanks chrome</h4>');
import_js('/.cpr/deps/' + fn, unmodal); import_js('/.cpr/deps/' + fn, unmodal);
@@ -509,7 +494,6 @@ function up2k_init(subtle) {
ebi('u2foot').innerHTML = 'seems like ' + shame + ' so do that if you want more performance <span style="color:#' + ebi('u2foot').innerHTML = 'seems like ' + shame + ' so do that if you want more performance <span style="color:#' +
(sha_js == 'ac' ? 'c84">(expecting 20' : '8a5">(but dont worry too much, expect 100') + ' MiB/s)</span>'; (sha_js == 'ac' ? 'c84">(expecting 20' : '8a5">(but dont worry too much, expect 100') + ' MiB/s)</span>';
} }
got_deps = true;
} }
// show uploader if the user only has write-access // show uploader if the user only has write-access
@@ -564,21 +548,17 @@ function up2k_init(subtle) {
ask_up = bcfg_get('ask_up', true), ask_up = bcfg_get('ask_up', true),
flag_en = bcfg_get('flag_en', false), flag_en = bcfg_get('flag_en', false),
fsearch = bcfg_get('fsearch', false), fsearch = bcfg_get('fsearch', false),
turbo = bcfg_get('u2turbo', false),
datechk = bcfg_get('u2tdate', true),
fdom_ctr = 0, fdom_ctr = 0,
min_filebuf = 0; min_filebuf = 0;
var st = { var st = {
"files": [], "files": [],
"todo": { "todo": {
"head": [],
"hash": [], "hash": [],
"handshake": [], "handshake": [],
"upload": [] "upload": []
}, },
"busy": { "busy": {
"head": [],
"hash": [], "hash": [],
"handshake": [], "handshake": [],
"upload": [] "upload": []
@@ -589,15 +569,6 @@ function up2k_init(subtle) {
} }
}; };
function push_t(arr, t) {
var sort = arr.length && arr[arr.length - 1].n > t.n;
arr.push(t);
if (sort)
arr.sort(function (a, b) {
return a.n < b.n ? -1 : 1;
});
}
var pvis = new U2pvis("bz", '#u2cards'); var pvis = new U2pvis("bz", '#u2cards');
var bobslice = null; var bobslice = null;
@@ -641,7 +612,7 @@ function up2k_init(subtle) {
} }
else files = e.target.files; else files = e.target.files;
if (!files || !files.length) if (!files || files.length == 0)
return alert('no files selected??'); return alert('no files selected??');
more_one_file(); more_one_file();
@@ -744,7 +715,8 @@ function up2k_init(subtle) {
pf.push(name); pf.push(name);
dn.file(function (fobj) { dn.file(function (fobj) {
apop(pf, name); var idx = pf.indexOf(name);
pf.splice(idx, 1);
try { try {
if (fobj.size > 0) { if (fobj.size > 0) {
good.push([fobj, name]); good.push([fobj, name]);
@@ -767,7 +739,7 @@ function up2k_init(subtle) {
} }
function gotallfiles(good_files, bad_files) { function gotallfiles(good_files, bad_files) {
if (bad_files.length) { if (bad_files.length > 0) {
var ntot = bad_files.length + good_files.length, var ntot = bad_files.length + good_files.length,
msg = 'These {0} files (of {1} total) were skipped because they are empty:\n'.format(bad_files.length, ntot); msg = 'These {0} files (of {1} total) were skipped because they are empty:\n'.format(bad_files.length, ntot);
@@ -825,10 +797,7 @@ function up2k_init(subtle) {
], fobj.size, draw_each); ], fobj.size, draw_each);
st.files.push(entry); st.files.push(entry);
if (turbo) st.todo.hash.push(entry);
push_t(st.todo.head, entry);
else
push_t(st.todo.hash, entry);
} }
if (!draw_each) { if (!draw_each) {
pvis.drawcard("q"); pvis.drawcard("q");
@@ -868,34 +837,17 @@ function up2k_init(subtle) {
// //
function handshakes_permitted() { function handshakes_permitted() {
if (!st.todo.handshake.length) var lim = multitask ? 1 : 0;
return true;
var t = st.todo.handshake[0], if (lim <
cd = t.cooldown;
if (cd && cd - Date.now() > 0)
return false;
// keepalive or verify
if (t.keepalive ||
t.t_uploaded)
return true;
if (parallel_uploads <
st.busy.handshake.length)
return false;
if (st.busy.handshake.length)
for (var n = t.n - 1; n >= t.n - parallel_uploads && n >= 0; n--)
if (st.files[n].t_uploading)
return false;
if ((multitask ? 1 : 0) <
st.todo.upload.length + st.todo.upload.length +
st.busy.upload.length) st.busy.upload.length)
return false; return false;
var cd = st.todo.handshake.length ? st.todo.handshake[0].cooldown : 0;
if (cd && cd - Date.now() > 0)
return false;
return true; return true;
} }
@@ -926,21 +878,15 @@ function up2k_init(subtle) {
return; return;
clearTimeout(tto); clearTimeout(tto);
if (crashed)
return defer();
running = true; running = true;
while (true) { while (window['vis_exh']) {
var now = Date.now(), var is_busy = 0 !=
is_busy = 0 != st.todo.hash.length +
st.todo.head.length + st.todo.handshake.length +
st.todo.hash.length + st.todo.upload.length +
st.todo.handshake.length + st.busy.hash.length +
st.todo.upload.length + st.busy.handshake.length +
st.busy.head.length + st.busy.upload.length;
st.busy.hash.length +
st.busy.handshake.length +
st.busy.upload.length;
if (was_busy != is_busy) { if (was_busy != is_busy) {
was_busy = is_busy; was_busy = is_busy;
@@ -951,6 +897,7 @@ function up2k_init(subtle) {
if (flag) { if (flag) {
if (is_busy) { if (is_busy) {
var now = Date.now();
flag.take(now); flag.take(now);
if (!flag.ours) if (!flag.ours)
return defer(); return defer();
@@ -962,57 +909,48 @@ function up2k_init(subtle) {
var mou_ikkai = false; var mou_ikkai = false;
if (st.busy.handshake.length && if (st.busy.handshake.length > 0 &&
st.busy.handshake[0].t_busied < now - 30 * 1000 st.busy.handshake[0].busied < Date.now() - 30 * 1000
) { ) {
console.log("retrying stuck handshake"); console.log("retrying stuck handshake");
var t = st.busy.handshake.shift(); var t = st.busy.handshake.shift();
st.todo.handshake.unshift(t); st.todo.handshake.unshift(t);
} }
var nprev = -1; if (st.todo.handshake.length > 0 &&
for (var a = 0; a < st.todo.upload.length; a++) { st.busy.handshake.length == 0 && (
var nf = st.todo.upload[a].nfile; st.todo.handshake[0].t4 || (
if (nprev == nf) handshakes_permitted() &&
continue; st.busy.upload.length < parallel_uploads
)
nprev = nf; )
var t = st.files[nf]; ) {
if (now - t.t_busied > 1000 * 30 &&
now - t.t_handshake > 1000 * (21600 - 1800)
) {
apop(st.todo.handshake, t);
st.todo.handshake.unshift(t);
t.keepalive = true;
}
}
if (st.todo.head.length &&
st.busy.head.length < parallel_uploads) {
exec_head();
mou_ikkai = true;
}
if (handshakes_permitted() &&
st.todo.handshake.length) {
exec_handshake(); exec_handshake();
mou_ikkai = true; mou_ikkai = true;
} }
if (st.todo.upload.length && if (handshakes_permitted() &&
st.todo.handshake.length > 0 &&
st.busy.handshake.length == 0 &&
st.busy.upload.length < parallel_uploads) {
exec_handshake();
mou_ikkai = true;
}
if (st.todo.upload.length > 0 &&
st.busy.upload.length < parallel_uploads) { st.busy.upload.length < parallel_uploads) {
exec_upload(); exec_upload();
mou_ikkai = true; mou_ikkai = true;
} }
if (hashing_permitted() && if (hashing_permitted() &&
st.todo.hash.length && st.todo.hash.length > 0 &&
!st.busy.hash.length) { st.busy.hash.length == 0) {
exec_hash(); exec_hash();
mou_ikkai = true; mou_ikkai = true;
} }
if (!mou_ikkai || crashed) if (!mou_ikkai)
return defer(); return defer();
} }
} }
@@ -1142,7 +1080,7 @@ function up2k_init(subtle) {
if (handled) { if (handled) {
pvis.move(t.n, 'ng'); pvis.move(t.n, 'ng');
apop(st.busy.hash, t); st.busy.hash.splice(st.busy.hash.indexOf(t), 1);
st.bytes.uploaded += t.size; st.bytes.uploaded += t.size;
return tasker(); return tasker();
} }
@@ -1175,11 +1113,15 @@ function up2k_init(subtle) {
t.hash.push(hashtab[a]); t.hash.push(hashtab[a]);
} }
t.t_hashed = Date.now(); t.t2 = Date.now();
if (t.n == 0 && window.location.hash == '#dbg') {
var spd = (t.size / ((t.t2 - t.t1) / 1000.)) / (1024 * 1024.);
alert('{0} ms, {1} MB/s\n'.format(t.t2 - t.t1, spd.toFixed(3)) + t.hash.join('\n'));
}
pvis.seth(t.n, 2, 'hashing done'); pvis.seth(t.n, 2, 'hashing done');
pvis.seth(t.n, 1, '📦 wait'); pvis.seth(t.n, 1, '📦 wait');
apop(st.busy.hash, t); st.busy.hash.splice(st.busy.hash.indexOf(t), 1);
st.todo.handshake.push(t); st.todo.handshake.push(t);
tasker(); tasker();
}; };
@@ -1202,57 +1144,10 @@ function up2k_init(subtle) {
}, 1); }, 1);
}; };
t.t_hashing = Date.now(); t.t1 = Date.now();
segm_next(); segm_next();
} }
/////
////
/// head
//
function exec_head() {
var t = st.todo.head.shift();
st.busy.head.push(t);
var xhr = new XMLHttpRequest();
xhr.onerror = function () {
console.log('head onerror, retrying', t);
apop(st.busy.head, t);
st.todo.head.unshift(t);
tasker();
};
function orz(e) {
var ok = false;
if (xhr.status == 200) {
var srv_sz = xhr.getResponseHeader('Content-Length'),
srv_ts = xhr.getResponseHeader('Last-Modified');
ok = t.size == srv_sz;
if (ok && datechk) {
srv_ts = new Date(srv_ts) / 1000;
ok = Math.abs(srv_ts - t.lmod) < 2;
}
}
apop(st.busy.head, t);
if (!ok)
return push_t(st.todo.hash, t);
t.done = true;
st.bytes.hashed += t.size;
st.bytes.uploaded += t.size;
pvis.seth(t.n, 1, 'YOLO');
pvis.seth(t.n, 2, "turbo'd");
pvis.move(t.n, 'ok');
};
xhr.onload = function (e) {
try { orz(e); } catch (ex) { vis_exh(ex + '', '', '', '', ex); }
};
xhr.open('HEAD', t.purl + t.name, true);
xhr.send();
}
///// /////
//// ////
/// handshake /// handshake
@@ -1260,48 +1155,37 @@ function up2k_init(subtle) {
function exec_handshake() { function exec_handshake() {
var t = st.todo.handshake.shift(), var t = st.todo.handshake.shift(),
keepalive = t.keepalive,
me = Date.now(); me = Date.now();
st.busy.handshake.push(t); st.busy.handshake.push(t);
t.keepalive = undefined; t.busied = me;
t.t_busied = me;
if (keepalive)
console.log("sending keepalive handshake", t);
var xhr = new XMLHttpRequest(); var xhr = new XMLHttpRequest();
xhr.onerror = function () { xhr.onerror = function () {
if (t.t_busied != me) { if (t.busied != me) {
console.log('zombie handshake onerror,', t); console.log('zombie handshake onerror,', t);
return; return;
} }
console.log('handshake onerror, retrying', t); console.log('handshake onerror, retrying', t);
apop(st.busy.handshake, t); st.busy.handshake.splice(st.busy.handshake.indexOf(t), 1);
st.todo.handshake.unshift(t); st.todo.handshake.unshift(t);
t.keepalive = keepalive;
tasker(); tasker();
}; };
function orz(e) { function orz(e) {
if (t.t_busied != me) { if (t.busied != me) {
console.log('zombie handshake onload,', t); console.log('zombie handshake onload,', t);
return; return;
} }
if (xhr.status == 200) { if (xhr.status == 200) {
t.t_handshake = Date.now();
if (keepalive) {
apop(st.busy.handshake, t);
return;
}
var response = JSON.parse(xhr.responseText); var response = JSON.parse(xhr.responseText);
if (!response.name) { if (!response.name) {
var msg = '', var msg = '',
smsg = ''; smsg = '';
if (!response || !response.hits || !response.hits.length) { if (!response || !response.hits || !response.hits.length) {
msg = 'not found on server';
smsg = '404'; smsg = '404';
msg = 'not found on server <a href="#" onclick="fsearch_explain()" class="fsearch_explain">(explain)</a>';
} }
else { else {
smsg = 'found'; smsg = 'found';
@@ -1318,7 +1202,7 @@ function up2k_init(subtle) {
pvis.seth(t.n, 2, msg); pvis.seth(t.n, 2, msg);
pvis.seth(t.n, 1, smsg); pvis.seth(t.n, 1, smsg);
pvis.move(t.n, smsg == '404' ? 'ng' : 'ok'); pvis.move(t.n, smsg == '404' ? 'ng' : 'ok');
apop(st.busy.handshake, t); st.busy.handshake.splice(st.busy.handshake.indexOf(t), 1);
st.bytes.uploaded += t.size; st.bytes.uploaded += t.size;
t.done = true; t.done = true;
tasker(); tasker();
@@ -1360,41 +1244,31 @@ function up2k_init(subtle) {
var done = true, var done = true,
msg = '&#x1f3b7;&#x1f41b;'; msg = '&#x1f3b7;&#x1f41b;';
if (t.postlist.length) { if (t.postlist.length > 0) {
var arr = st.todo.upload,
sort = arr.length && arr[arr.length - 1].nfile > t.n;
for (var a = 0; a < t.postlist.length; a++) for (var a = 0; a < t.postlist.length; a++)
arr.push({ st.todo.upload.push({
'nfile': t.n, 'nfile': t.n,
'npart': t.postlist[a] 'npart': t.postlist[a]
}); });
msg = 'uploading'; msg = 'uploading';
done = false; done = false;
if (sort)
arr.sort(function (a, b) {
return a.nfile < b.nfile ? -1 :
/* */ a.nfile > b.nfile ? 1 :
a.npart < b.npart ? -1 : 1;
});
} }
pvis.seth(t.n, 1, msg); pvis.seth(t.n, 1, msg);
apop(st.busy.handshake, t); st.busy.handshake.splice(st.busy.handshake.indexOf(t), 1);
if (done) { if (done) {
t.done = true; t.done = true;
st.bytes.uploaded += t.size - t.bytes_uploaded; st.bytes.uploaded += t.size - t.bytes_uploaded;
var spd1 = (t.size / ((t.t_hashed - t.t_hashing) / 1000.)) / (1024 * 1024.), var spd1 = (t.size / ((t.t2 - t.t1) / 1000.)) / (1024 * 1024.),
spd2 = (t.size / ((t.t_uploaded - t.t_uploading) / 1000.)) / (1024 * 1024.); spd2 = (t.size / ((t.t4 - t.t3) / 1000.)) / (1024 * 1024.);
pvis.seth(t.n, 2, 'hash {0}, up {1} MB/s'.format( pvis.seth(t.n, 2, 'hash {0}, up {1} MB/s'.format(
spd1.toFixed(2), spd2.toFixed(2))); spd1.toFixed(2), spd2.toFixed(2)));
pvis.move(t.n, 'ok'); pvis.move(t.n, 'ok');
} }
else t.t_uploaded = undefined; else t.t4 = undefined;
tasker(); tasker();
} }
@@ -1413,7 +1287,7 @@ function up2k_init(subtle) {
var penalty = rsp.replace(/.*rate-limit /, "").split(' ')[0]; var penalty = rsp.replace(/.*rate-limit /, "").split(' ')[0];
console.log("rate-limit: " + penalty); console.log("rate-limit: " + penalty);
t.cooldown = Date.now() + parseFloat(penalty) * 1000; t.cooldown = Date.now() + parseFloat(penalty) * 1000;
apop(st.busy.handshake, t); st.busy.handshake.splice(st.busy.handshake.indexOf(t), 1);
st.todo.handshake.unshift(t); st.todo.handshake.unshift(t);
return; return;
} }
@@ -1432,7 +1306,7 @@ function up2k_init(subtle) {
pvis.seth(t.n, 2, err); pvis.seth(t.n, 2, err);
pvis.move(t.n, 'ng'); pvis.move(t.n, 'ng');
apop(st.busy.handshake, t); st.busy.handshake.splice(st.busy.handshake.indexOf(t), 1);
tasker(); tasker();
return; return;
} }
@@ -1473,8 +1347,8 @@ function up2k_init(subtle) {
var npart = upt.npart, var npart = upt.npart,
t = st.files[upt.nfile]; t = st.files[upt.nfile];
if (!t.t_uploading) if (!t.t3)
t.t_uploading = Date.now(); t.t3 = Date.now();
pvis.seth(t.n, 1, "🚀 send"); pvis.seth(t.n, 1, "🚀 send");
@@ -1493,17 +1367,17 @@ function up2k_init(subtle) {
t.bytes_uploaded += cdr - car; t.bytes_uploaded += cdr - car;
} }
else if (txt.indexOf('already got that') !== -1) { else if (txt.indexOf('already got that') !== -1) {
console.log("ignoring dupe-segment error", t); console.log("ignoring dupe-segment error");
} }
else { else {
alert("server broke; cu-err {0} on file [{1}]:\n".format( alert("server broke; cu-err {0} on file [{1}]:\n".format(
xhr.status, t.name) + (txt || "no further information")); xhr.status, t.name) + (txt || "no further information"));
return; return;
} }
apop(st.busy.upload, upt); st.busy.upload.splice(st.busy.upload.indexOf(upt), 1);
apop(t.postlist, npart); t.postlist.splice(t.postlist.indexOf(npart), 1);
if (!t.postlist.length) { if (t.postlist.length == 0) {
t.t_uploaded = Date.now(); t.t4 = Date.now();
pvis.seth(t.n, 1, 'verifying'); pvis.seth(t.n, 1, 'verifying');
st.todo.handshake.unshift(t); st.todo.handshake.unshift(t);
} }
@@ -1518,7 +1392,7 @@ function up2k_init(subtle) {
try { orz(xhr); } catch (ex) { vis_exh(ex + '', '', '', '', ex); } try { orz(xhr); } catch (ex) { vis_exh(ex + '', '', '', '', ex); }
}; };
xhr.onerror = function (xev) { xhr.onerror = function (xev) {
if (crashed) if (!window['vis_exh'])
return; return;
console.log('chunkpit onerror, retrying', t); console.log('chunkpit onerror, retrying', t);
@@ -1572,7 +1446,7 @@ function up2k_init(subtle) {
for (var a = o.length - 1; a >= 0; a--) { for (var a = o.length - 1; a >= 0; a--) {
o[a].parentNode.getElementsByTagName('input')[0].setAttribute('tt', o[a].getAttribute('tt')); o[a].parentNode.getElementsByTagName('input')[0].setAttribute('tt', o[a].getAttribute('tt'));
} }
tt.att(QS('#u2conf')); tt.init();
function bumpthread2(e) { function bumpthread2(e) {
if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing) if (e.ctrlKey || e.altKey || e.metaKey || e.isComposing)
@@ -1630,35 +1504,6 @@ function up2k_init(subtle) {
set_fsearch(!fsearch); set_fsearch(!fsearch);
} }
function tgl_turbo() {
turbo = !turbo;
bcfg_set('u2turbo', turbo);
draw_turbo();
}
function tgl_datechk() {
datechk = !datechk;
bcfg_set('u2tdate', datechk);
}
function draw_turbo() {
var msgu = '<p class="warn">WARNING: turbo enabled, <span>&nbsp;client may not detect and resume incomplete uploads; see turbo-button tooltip</span></p>',
msgs = '<p class="warn">WARNING: turbo enabled, <span>&nbsp;search may give false-positives; see turbo-button tooltip</span></p>',
msg = fsearch ? msgs : msgu,
omsg = fsearch ? msgu : msgs,
html = ebi('u2foot').innerHTML,
ohtml = html;
if (turbo && html.indexOf(msg) === -1)
html = html.replace(omsg, '') + msg;
else if (!turbo)
html = html.replace(msgu, '').replace(msgs, '');
if (html !== ohtml)
ebi('u2foot').innerHTML = html;
}
draw_turbo();
function set_fsearch(new_state) { function set_fsearch(new_state) {
var fixed = false; var fixed = false;
@@ -1696,7 +1541,6 @@ function up2k_init(subtle) {
} }
catch (ex) { } catch (ex) { }
draw_turbo();
onresize(); onresize();
} }
@@ -1741,8 +1585,6 @@ function up2k_init(subtle) {
ebi('multitask').addEventListener('click', tgl_multitask, false); ebi('multitask').addEventListener('click', tgl_multitask, false);
ebi('ask_up').addEventListener('click', tgl_ask_up, false); ebi('ask_up').addEventListener('click', tgl_ask_up, false);
ebi('flag_en').addEventListener('click', tgl_flag_en, false); ebi('flag_en').addEventListener('click', tgl_flag_en, false);
ebi('u2turbo').addEventListener('click', tgl_turbo, false);
ebi('u2tdate').addEventListener('click', tgl_datechk, false);
var o = ebi('fsearch'); var o = ebi('fsearch');
if (o) if (o)
o.addEventListener('click', tgl_fsearch, false); o.addEventListener('click', tgl_fsearch, false);

View File

@@ -87,9 +87,8 @@
#u2tab td:nth-child(3) { #u2tab td:nth-child(3) {
width: 40%; width: 40%;
} }
#op_up2k.srch td.prog { #op_up2k.srch #u2tab td:nth-child(3) {
font-family: sans-serif; font-family: sans-serif;
font-size: 1em;
width: auto; width: auto;
} }
#u2tab tbody tr:hover td { #u2tab tbody tr:hover td {
@@ -216,37 +215,15 @@
color: #fff; color: #fff;
font-style: italic; font-style: italic;
} }
#u2foot .warn {
font-size: 1.3em;
padding: .5em .8em;
margin: 1em -.6em;
color: #f74;
background: #322;
border: 1px solid #633;
border-width: .1em 0;
text-align: center;
}
#u2foot .warn span {
color: #f86;
}
html.light #u2foot .warn {
color: #b00;
background: #fca;
border-color: #f70;
}
html.light #u2foot .warn span {
color: #930;
}
#u2foot span { #u2foot span {
color: #999; color: #999;
font-size: .9em; font-size: .9em;
font-weight: normal;
} }
#u2footfoot { #u2footfoot {
margin-bottom: -1em; margin-bottom: -1em;
} }
.prog { .prog {
font-family: monospace, monospace; font-family: monospace;
} }
#u2tab a>span { #u2tab a>span {
font-weight: bold; font-weight: bold;
@@ -258,11 +235,6 @@ html.light #u2foot .warn span {
float: right; float: right;
margin-bottom: -.3em; margin-bottom: -.3em;
} }
.fsearch_explain {
padding-left: .7em;
font-size: 1.1em;
line-height: 0;
}

View File

@@ -7,14 +7,7 @@ if (!window['console'])
var is_touch = 'ontouchstart' in window, var is_touch = 'ontouchstart' in window,
IPHONE = /iPhone|iPad|iPod/i.test(navigator.userAgent), ANDROID = /(android)/i.test(navigator.userAgent);
ANDROID = /android/i.test(navigator.userAgent);
var ebi = document.getElementById.bind(document),
QS = document.querySelector.bind(document),
QSA = document.querySelectorAll.bind(document),
mknod = document.createElement.bind(document);
// error handler for mobile devices // error handler for mobile devices
@@ -28,65 +21,36 @@ function esc(txt) {
}[c]; }[c];
}); });
} }
var crashed = false, ignexd = {};
function vis_exh(msg, url, lineNo, columnNo, error) { function vis_exh(msg, url, lineNo, columnNo, error) {
if ((msg + '').indexOf('ResizeObserver') !== -1) if (!window.onerror)
return; // chrome issue 809574 (benign, from <video>)
var ekey = url + '\n' + lineNo + '\n' + msg;
if (ignexd[ekey] || crashed)
return; return;
crashed = true;
window.onerror = undefined; window.onerror = undefined;
var html = ['<h1>you hit a bug!</h1><p style="font-size:1.3em;margin:0">try to <a href="#" onclick="localStorage.clear();location.reload();">reset copyparty settings</a> if you are stuck here, or <a href="#" onclick="ignex();">ignore this</a> / <a href="#" onclick="ignex(true);">ignore all</a></p><p>please send me a screenshot arigathanks gozaimuch: <code>ed/irc.rizon.net</code> or <code>ed#2644</code><br />&nbsp; (and if you can, press F12 and include the "Console" tab in the screenshot too)</p><p>', window['vis_exh'] = null;
esc(url + ' @' + lineNo + ':' + columnNo), '<br />' + esc(String(msg)) + '</p>']; var html = ['<h1>you hit a bug!</h1><p>please send me a screenshot arigathanks gozaimuch: <code>ed/irc.rizon.net</code> or <code>ed#2644</code><br />&nbsp; (and if you can, press F12 and include the "Console" tab in the screenshot too)</p><p>',
esc(String(msg)), '</p><p>', esc(url + ' @' + lineNo + ':' + columnNo), '</p>'];
try { if (error) {
if (error) { var find = ['desc', 'stack', 'trace'];
var find = ['desc', 'stack', 'trace']; for (var a = 0; a < find.length; a++)
for (var a = 0; a < find.length; a++) if (String(error[find[a]]) !== 'undefined')
if (String(error[find[a]]) !== 'undefined') html.push('<h2>' + find[a] + '</h2>' +
html.push('<h3>' + find[a] + '</h3>' + esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
esc(String(error[find[a]])).replace(/\n/g, '<br />\n'));
}
ignexd[ekey] = true;
html.push('<h3>localStore</h3>' + esc(JSON.stringify(localStorage)));
} }
catch (e) { } document.body.innerHTML = html.join('\n');
try { var s = mknod('style');
var exbox = ebi('exbox'); s.innerHTML = 'body{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em} code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} *{line-height:1.5em}';
if (!exbox) { document.head.appendChild(s);
exbox = mknod('div');
exbox.setAttribute('id', 'exbox');
document.body.appendChild(exbox);
var s = mknod('style');
s.innerHTML = '#exbox{background:#333;color:#ddd;font-family:sans-serif;font-size:0.8em;padding:0 1em 1em 1em;z-index:80386;position:fixed;top:0;left:0;right:0;bottom:0;width:100%;height:100%} #exbox h1{margin:.5em 1em 0 0;padding:0} #exbox h3{border-top:1px solid #999;margin:1em 0 0 0} #exbox a{text-decoration:underline;color:#fc0} #exbox code{color:#bf7;background:#222;padding:.1em;margin:.2em;font-size:1.1em;font-family:monospace,monospace} #exbox *{line-height:1.5em}';
document.head.appendChild(s);
}
exbox.innerHTML = html.join('\n');
exbox.style.display = 'block';
}
catch (e) {
document.body.innerHTML = html.join('\n');
}
throw 'fatal_err'; throw 'fatal_err';
} }
function ignex(all) {
var o = ebi('exbox');
o.style.display = 'none';
o.innerHTML = '';
crashed = false;
if (!all)
window.onerror = vis_exh;
}
function ctrl(e) { var ebi = document.getElementById.bind(document),
return e && (e.ctrlKey || e.metaKey); QS = document.querySelector.bind(document),
} QSA = document.querySelectorAll.bind(document),
mknod = document.createElement.bind(document);
function ev(e) { function ev(e) {
@@ -123,15 +87,6 @@ if (!String.startsWith) {
return this.substring(i, i + s.length) === s; return this.substring(i, i + s.length) === s;
}; };
} }
if (!Element.prototype.closest) {
Element.prototype.closest = function (s) {
var el = this;
do {
if (el.msMatchesSelector(s)) return el;
el = el.parentElement || el.parentNode;
} while (el !== null && el.nodeType === 1);
}
}
// https://stackoverflow.com/a/950146 // https://stackoverflow.com/a/950146
@@ -361,18 +316,6 @@ function linksplit(rp) {
} }
function vsplit(vp) {
if (vp.endsWith('/'))
vp = vp.slice(0, -1);
var ofs = vp.lastIndexOf('/') + 1,
base = vp.slice(0, ofs),
fn = vp.slice(ofs);
return [base, fn];
}
function uricom_enc(txt, do_fb_enc) { function uricom_enc(txt, do_fb_enc) {
try { try {
return encodeURIComponent(txt); return encodeURIComponent(txt);
@@ -446,27 +389,25 @@ function has(haystack, needle) {
} }
function apop(arr, v) {
var ofs = arr.indexOf(v);
if (ofs !== -1)
arr.splice(ofs, 1);
}
function jcp(obj) { function jcp(obj) {
return JSON.parse(JSON.stringify(obj)); return JSON.parse(JSON.stringify(obj));
} }
function sread(key) { function sread(key) {
return localStorage.getItem(key); if (window.localStorage)
return localStorage.getItem(key);
return null;
} }
function swrite(key, val) { function swrite(key, val) {
if (val === undefined || val === null) if (window.localStorage) {
localStorage.removeItem(key); if (val === undefined || val === null)
else localStorage.removeItem(key);
localStorage.setItem(key, val); else
localStorage.setItem(key, val);
}
} }
function jread(key, fb) { function jread(key, fb) {
@@ -549,20 +490,13 @@ function hist_replace(url) {
var tt = (function () { var tt = (function () {
var r = { var r = {
"tt": mknod("div"), "tt": mknod("div"),
"en": true, "en": true
"el": null,
"skip": false
}; };
r.tt.setAttribute('id', 'tt'); r.tt.setAttribute('id', 'tt');
document.body.appendChild(r.tt); document.body.appendChild(r.tt);
r.show = function () { function show() {
if (r.skip) {
r.skip = false;
return;
}
var cfg = sread('tooltips'); var cfg = sread('tooltips');
if (cfg !== null && cfg != '1') if (cfg !== null && cfg != '1')
return; return;
@@ -571,62 +505,21 @@ var tt = (function () {
if (!msg) if (!msg)
return; return;
r.el = this;
var pos = this.getBoundingClientRect(), var pos = this.getBoundingClientRect(),
dir = this.getAttribute('ttd') || '',
left = pos.left < window.innerWidth / 2, left = pos.left < window.innerWidth / 2,
top = pos.top < window.innerHeight / 2, top = pos.top < window.innerHeight / 2;
big = this.className.indexOf(' ttb') !== -1;
if (dir.indexOf('u') + 1) top = false;
if (dir.indexOf('d') + 1) top = true;
if (dir.indexOf('l') + 1) left = false;
if (dir.indexOf('r') + 1) left = true;
clmod(r.tt, 'b', big);
r.tt.style.top = top ? pos.bottom + 'px' : 'auto'; r.tt.style.top = top ? pos.bottom + 'px' : 'auto';
r.tt.style.bottom = top ? 'auto' : (window.innerHeight - pos.top) + 'px'; r.tt.style.bottom = top ? 'auto' : (window.innerHeight - pos.top) + 'px';
r.tt.style.left = left ? pos.left + 'px' : 'auto'; r.tt.style.left = left ? pos.left + 'px' : 'auto';
r.tt.style.right = left ? 'auto' : (window.innerWidth - pos.right) + 'px'; r.tt.style.right = left ? 'auto' : (window.innerWidth - pos.right) + 'px';
r.tt.innerHTML = msg.replace(/\$N/g, "<br />"); r.tt.innerHTML = msg.replace(/\$N/g, "<br />");
r.el.addEventListener('mouseleave', r.hide);
clmod(r.tt, 'show', 1); clmod(r.tt, 'show', 1);
};
r.hide = function (e) {
ev(e);
clmod(r.tt, 'show');
if (r.el)
r.el.removeEventListener('mouseleave', r.hide);
};
if (is_touch && IPHONE) {
var f1 = r.show,
f2 = r.hide;
r.show = function () {
setTimeout(f1.bind(this), 301);
};
r.hide = function () {
setTimeout(f2.bind(this), 301);
};
} }
r.tt.onclick = r.hide; function hide() {
clmod(r.tt, 'show');
r.att = function (ctr) {
var _show = r.en ? r.show : null,
_hide = r.en ? r.hide : null,
o = ctr.querySelectorAll('*[tt]');
for (var a = o.length - 1; a >= 0; a--) {
o[a].onfocus = _show;
o[a].onblur = _hide;
o[a].onmouseenter = _show;
o[a].onmouseleave = _hide;
}
r.hide();
} }
r.init = function () { r.init = function () {
@@ -640,58 +533,18 @@ var tt = (function () {
}; };
r.en = bcfg_get('tooltips', true) r.en = bcfg_get('tooltips', true)
} }
r.att(document);
}; var _show = r.en ? show : null,
_hide = r.en ? hide : null;
return r;
})(); var o = QSA('*[tt]');
for (var a = o.length - 1; a >= 0; a--) {
o[a].onfocus = _show;
var toast = (function () { o[a].onblur = _hide;
var r = {}, o[a].onmouseenter = _show;
te = null, o[a].onmouseleave = _hide;
visible = false, }
obj = mknod('div'); hide();
obj.setAttribute('id', 'toast');
document.body.appendChild(obj);;
r.hide = function (e) {
ev(e);
clearTimeout(te);
clmod(obj, 'vis');
r.visible = false;
};
r.show = function (cl, ms, txt) {
clearTimeout(te);
if (ms)
te = setTimeout(r.hide, ms * 1000);
var html = '', hp = txt.split(/(?=<.?pre>)/i);
for (var a = 0; a < hp.length; a++)
html += hp[a].startsWith('<pre>') ? hp[a] :
hp[a].replace(/<br ?.?>\n/g, '\n').replace(/\n<br ?.?>/g, '\n').replace(/\n/g, '<br />\n');
obj.innerHTML = '<a href="#" id="toastc">x</a>' + html;
obj.className = cl;
ms += obj.offsetWidth;
obj.className += ' vis';
ebi('toastc').onclick = r.hide;
r.visible = true;
};
r.ok = function (ms, txt) {
r.show('ok', ms, txt);
};
r.inf = function (ms, txt) {
r.show('inf', ms, txt);
};
r.warn = function (ms, txt) {
r.show('warn', ms, txt);
};
r.err = function (ms, txt) {
r.show('err', ms, txt);
}; };
return r; return r;

View File

@@ -10,25 +10,19 @@ u k:k
# share "." (the current directory) # share "." (the current directory)
# as "/" (the webroot) for the following users: # as "/" (the webroot) for the following users:
# "r" grants read-access for anyone # "r" grants read-access for anyone
# "rw ed" grants read-write to ed # "a ed" grants read-write to ed
. .
/ /
r r
rw ed a ed
# custom permissions for the "priv" folder: # custom permissions for the "priv" folder:
# user "k" can only see/read the contents # user "k" can see/read the contents
# user "ed" gets read-write access # and "ed" gets read-write access
./priv ./priv
/priv /priv
r k r k
rw ed a ed
# this does the same thing:
./priv
/priv
r ed k
w ed
# share /home/ed/Music/ as /music and let anyone read it # share /home/ed/Music/ as /music and let anyone read it
# (this will replace any folder called "music" in the webroot) # (this will replace any folder called "music" in the webroot)
@@ -47,5 +41,5 @@ c e2d
c nodupe c nodupe
# this entire config file can be replaced with these arguments: # this entire config file can be replaced with these arguments:
# -u ed:123 -u k:k -v .::r:a,ed -v priv:priv:r,k:rw,ed -v /home/ed/Music:music:r -v /home/ed/inc:dump:w:c,e2d:c,nodupe # -u ed:123 -u k:k -v .::r:aed -v priv:priv:rk:aed -v /home/ed/Music:music:r -v /home/ed/inc:dump:w
# but note that the config file always wins in case of conflicts # but note that the config file always wins in case of conflicts

View File

@@ -1,51 +0,0 @@
<!DOCTYPE html><html lang="en"><head>
<meta charset="utf-8">
<title>hls-test</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
</head><body>
<video id="vid" controls></video>
<script src="hls.light.js"></script>
<script>
var video = document.getElementById('vid');
var hls = new Hls({
debug: true,
autoStartLoad: false
});
hls.loadSource('live/v.m3u8');
hls.attachMedia(video);
hls.on(Hls.Events.MANIFEST_PARSED, function() {
hls.startLoad(0);
});
hls.on(Hls.Events.MEDIA_ATTACHED, function() {
video.muted = true;
video.play();
});
/*
general good news:
- doesn't need fixed-length segments; ok to let x264 pick optimal keyframes and slice on those
- hls.js polls the m3u8 for new segments, scales the duration accordingly, seeking works great
- the sfx will grow by 66 KiB since that's how small hls.js can get, wait thats not good
# vod, creates m3u8 at the end, fixed keyframes, v bad
ffmpeg -hide_banner -threads 0 -flags -global_header -i ..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -g 120 -keyint_min 120 -sc_threshold 0 -hls_time 4 -hls_playlist_type vod -hls_segment_filename v%05d.ts v.m3u8
# live, updates m3u8 as it goes, dynamic keyframes, streamable with hls.js
ffmpeg -hide_banner -threads 0 -flags -global_header -i ..\..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -f segment -segment_list v.m3u8 -segment_format mpegts -segment_list_flags live v%05d.ts
# fmp4 (fragmented mp4), doesn't work with hls.js, gets duratoin 149:07:51 (536871s), probably the tkhd/mdhd 0xffffffff (timebase 8000? ok)
ffmpeg -re -hide_banner -threads 0 -flags +cgop -i ..\..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -f segment -segment_list v.m3u8 -segment_format fmp4 -segment_list_flags live v%05d.mp4
# try 2, works, uses tempfiles for m3u8 updates, good, 6% smaller
ffmpeg -re -hide_banner -threads 0 -flags +cgop -i ..\..\CowboyBebopMovie-OP1.webm -vf scale=1280:-4,format=yuv420p -ac 2 -c:a libopus -b:a 128k -c:v libx264 -preset slow -crf 24 -maxrate:v 5M -bufsize:v 10M -f hls -hls_segment_type fmp4 -hls_list_size 0 -hls_segment_filename v%05d.mp4 v.m3u8
more notes
- adding -hls_flags single_file makes duration wack during playback (for both fmp4 and ts), ok once finalized and refreshed, gives no size reduction anyways
- bebop op has good keyframe spacing for testing hls.js, in particular it hops one seg back and immediately resumes if it hits eof with the explicit hls.startLoad(0); otherwise it jumps into the middle of a seg and becomes art
- can probably -c:v copy most of the time, is there a way to check for cgop? todo
*/
</script>
</body></html>

View File

@@ -166,10 +166,7 @@ dbg.asyncStore.pendingBreakpoints = {}
about:config >> devtools.debugger.prefs-schema-version = -1 about:config >> devtools.debugger.prefs-schema-version = -1
# determine server version # determine server version
git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --decorate=full > ../revs && cat ../{util,browser,up2k}.js >../vr && cat ../revs | while read -r rev extra; do (git reset --hard $rev >/dev/null 2>/dev/null && dsz=$(cat copyparty/web/{util,browser,up2k}.js >../vg 2>/dev/null && diff -wNarU0 ../{vg,vr} | wc -c) && printf '%s %6s %s\n' "$rev" $dsz "$extra") </dev/null; done git pull; git reset --hard origin/HEAD && git log --format=format:"%H %ai %d" --decorate=full > ../revs && cat ../{util,browser}.js >../vr && cat ../revs | while read -r rev extra; do (git reset --hard $rev >/dev/null 2>/dev/null && dsz=$(cat copyparty/web/{util,browser}.js >../vg 2>/dev/null && diff -wNarU0 ../{vg,vr} | wc -c) && printf '%s %6s %s\n' "$rev" $dsz "$extra") </dev/null; done
# download all sfx versions
curl https://api.github.com/repos/9001/copyparty/releases?per_page=100 | jq -r '.[] | .tag_name + " " + .name' | while read v t; do fn="copyparty $v $t.py"; [ -e $fn ] || curl https://github.com/9001/copyparty/releases/download/$v/copyparty-sfx.py -Lo "$fn"; done
## ##

View File

@@ -20,11 +20,6 @@ echo
# #
# `no-cm` saves ~90k by removing easymde/codemirror # `no-cm` saves ~90k by removing easymde/codemirror
# (the fancy markdown editor) # (the fancy markdown editor)
#
# `no-fnt` saves ~9k by removing the source-code-pro font
# (mainly used my the markdown viewer/editor)
#
# `no-dd` saves ~2k by removing the mouse cursor
# port install gnutar findutils gsed coreutils # port install gnutar findutils gsed coreutils
@@ -62,18 +57,14 @@ use_gz=
do_sh=1 do_sh=1
do_py=1 do_py=1
while [ ! -z "$1" ]; do while [ ! -z "$1" ]; do
case $1 in [ "$1" = clean ] && clean=1 && shift && continue
clean) clean=1 ; ;; [ "$1" = re ] && repack=1 && shift && continue
re) repack=1 ; ;; [ "$1" = gz ] && use_gz=1 && shift && continue
gz) use_gz=1 ; ;; [ "$1" = no-ogv ] && no_ogv=1 && shift && continue
no-ogv) no_ogv=1 ; ;; [ "$1" = no-cm ] && no_cm=1 && shift && continue
no-fnt) no_fnt=1 ; ;; [ "$1" = no-sh ] && do_sh= && shift && continue
no-dd) no_dd=1 ; ;; [ "$1" = no-py ] && do_py= && shift && continue
no-cm) no_cm=1 ; ;; break
no-sh) do_sh= ; ;;
no-py) do_py= ; ;;
esac
shift
done done
tmv() { tmv() {
@@ -199,18 +190,6 @@ done
sed -r '/edit2">edit \(fancy/d' <$f >t && tmv "$f" sed -r '/edit2">edit \(fancy/d' <$f >t && tmv "$f"
} }
[ $no_fnt ] && {
rm -f copyparty/web/deps/scp.woff2
f=copyparty/web/md.css
sed -r '/scp\.woff2/d' <$f >t && tmv "$f"
}
[ $no_dd ] && {
rm -rf copyparty/web/dd
f=copyparty/web/browser.css
sed -r 's/(cursor: )url\([^)]+\), (pointer)/\1\2/; /[0-9]+% \{cursor:/d; /animation: cursor/d' <$f >t && tmv "$f"
}
[ $repack ] || [ $repack ] ||
find | grep -E '\.py$' | find | grep -E '\.py$' |
grep -vE '__version__' | grep -vE '__version__' |

View File

@@ -6,8 +6,8 @@ import re, os, sys, time, shutil, signal, threading, tarfile, hashlib, platform,
import subprocess as sp import subprocess as sp
""" """
to edit this file, use HxD or "vim -b" pls don't edit this file with a text editor,
(there is compressed stuff at the end) it breaks the compressed stuff at the end
run me with any version of python, i will unpack and run copyparty run me with any version of python, i will unpack and run copyparty
@@ -380,7 +380,7 @@ def run(tmp, j2):
fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
except Exception as ex: except Exception as ex:
if not WINDOWS: if not WINDOWS:
msg("\033[31mflock:{!r}\033[0m".format(ex)) msg("\033[31mflock:", repr(ex))
t = threading.Thread(target=utime, args=(tmp,)) t = threading.Thread(target=utime, args=(tmp,))
t.daemon = True t.daemon = True

View File

@@ -23,18 +23,14 @@ def hdr(query):
class Cfg(Namespace): class Cfg(Namespace):
def __init__(self, a=None, v=None, c=None): def __init__(self, a=[], v=[], c=None):
super(Cfg, self).__init__( super(Cfg, self).__init__(
a=a or [], a=a,
v=v or [], v=v,
c=c, c=c,
rproxy=0, rproxy=0,
ed=False, ed=False,
nw=False,
no_mv=False,
no_del=False,
no_zip=False, no_zip=False,
no_voldump=True,
no_scandir=False, no_scandir=False,
no_sendfile=True, no_sendfile=True,
no_rescan=True, no_rescan=True,
@@ -93,7 +89,7 @@ class TestHttpCli(unittest.TestCase):
if not vol.startswith(top): if not vol.startswith(top):
continue continue
mode = vol[-2].replace("a", "rwmd") mode = vol[-2]
usr = vol[-1] usr = vol[-1]
if usr == "a": if usr == "a":
usr = "" usr = ""
@@ -102,7 +98,7 @@ class TestHttpCli(unittest.TestCase):
vol += "/" vol += "/"
top, sub = vol.split("/", 1) top, sub = vol.split("/", 1)
vcfg.append("{0}/{1}:{1}:{2},{3}".format(top, sub, mode, usr)) vcfg.append("{0}/{1}:{1}:{2}{3}".format(top, sub, mode, usr))
pprint.pprint(vcfg) pprint.pprint(vcfg)

View File

@@ -16,19 +16,18 @@ from copyparty import util
class Cfg(Namespace): class Cfg(Namespace):
def __init__(self, a=None, v=None, c=None): def __init__(self, a=[], v=[], c=None):
ex = {k: False for k in "nw e2d e2ds e2dsa e2t e2ts e2tsr".split()} ex = {k: False for k in "e2d e2ds e2dsa e2t e2ts e2tsr".split()}
ex2 = { ex2 = {
"mtp": [], "mtp": [],
"mte": "a", "mte": "a",
"hist": None, "hist": None,
"no_hash": False, "no_hash": False,
"css_browser": None, "css_browser": None,
"no_voldump": True,
"rproxy": 0, "rproxy": 0,
} }
ex.update(ex2) ex.update(ex2)
super(Cfg, self).__init__(a=a or [], v=v or [], c=c, **ex) super(Cfg, self).__init__(a=a, v=v, c=c, **ex)
class TestVFS(unittest.TestCase): class TestVFS(unittest.TestCase):
@@ -58,8 +57,8 @@ class TestVFS(unittest.TestCase):
# type: (VFS, str, str) -> tuple[str, str, str] # type: (VFS, str, str) -> tuple[str, str, str]
"""helper for resolving and listing a folder""" """helper for resolving and listing a folder"""
vn, rem = vfs.get(vpath, uname, True, False) vn, rem = vfs.get(vpath, uname, True, False)
r1 = vn.ls(rem, uname, False, [[True]]) r1 = vn.ls(rem, uname, False)
r2 = vn.ls(rem, uname, False, [[True]]) r2 = vn.ls(rem, uname, False)
self.assertEqual(r1, r2) self.assertEqual(r1, r2)
fsdir, real, virt = r1 fsdir, real, virt = r1
@@ -69,11 +68,6 @@ class TestVFS(unittest.TestCase):
def log(self, src, msg, c=0): def log(self, src, msg, c=0):
pass pass
def assertAxs(self, dct, lst):
t1 = list(sorted(dct.keys()))
t2 = list(sorted(lst))
self.assertEqual(t1, t2)
def test(self): def test(self):
td = os.path.join(self.td, "vfs") td = os.path.join(self.td, "vfs")
os.mkdir(td) os.mkdir(td)
@@ -94,53 +88,53 @@ class TestVFS(unittest.TestCase):
self.assertEqual(vfs.nodes, {}) self.assertEqual(vfs.nodes, {})
self.assertEqual(vfs.vpath, "") self.assertEqual(vfs.vpath, "")
self.assertEqual(vfs.realpath, td) self.assertEqual(vfs.realpath, td)
self.assertAxs(vfs.axs.uread, ["*"]) self.assertEqual(vfs.uread, ["*"])
self.assertAxs(vfs.axs.uwrite, ["*"]) self.assertEqual(vfs.uwrite, ["*"])
# single read-only rootfs (relative path) # single read-only rootfs (relative path)
vfs = AuthSrv(Cfg(v=["a/ab/::r"]), self.log).vfs vfs = AuthSrv(Cfg(v=["a/ab/::r"]), self.log).vfs
self.assertEqual(vfs.nodes, {}) self.assertEqual(vfs.nodes, {})
self.assertEqual(vfs.vpath, "") self.assertEqual(vfs.vpath, "")
self.assertEqual(vfs.realpath, os.path.join(td, "a", "ab")) self.assertEqual(vfs.realpath, os.path.join(td, "a", "ab"))
self.assertAxs(vfs.axs.uread, ["*"]) self.assertEqual(vfs.uread, ["*"])
self.assertAxs(vfs.axs.uwrite, []) self.assertEqual(vfs.uwrite, [])
# single read-only rootfs (absolute path) # single read-only rootfs (absolute path)
vfs = AuthSrv(Cfg(v=[td + "//a/ac/../aa//::r"]), self.log).vfs vfs = AuthSrv(Cfg(v=[td + "//a/ac/../aa//::r"]), self.log).vfs
self.assertEqual(vfs.nodes, {}) self.assertEqual(vfs.nodes, {})
self.assertEqual(vfs.vpath, "") self.assertEqual(vfs.vpath, "")
self.assertEqual(vfs.realpath, os.path.join(td, "a", "aa")) self.assertEqual(vfs.realpath, os.path.join(td, "a", "aa"))
self.assertAxs(vfs.axs.uread, ["*"]) self.assertEqual(vfs.uread, ["*"])
self.assertAxs(vfs.axs.uwrite, []) self.assertEqual(vfs.uwrite, [])
# read-only rootfs with write-only subdirectory (read-write for k) # read-only rootfs with write-only subdirectory (read-write for k)
vfs = AuthSrv( vfs = AuthSrv(
Cfg(a=["k:k"], v=[".::r:rw,k", "a/ac/acb:a/ac/acb:w:rw,k"]), Cfg(a=["k:k"], v=[".::r:ak", "a/ac/acb:a/ac/acb:w:ak"]),
self.log, self.log,
).vfs ).vfs
self.assertEqual(len(vfs.nodes), 1) self.assertEqual(len(vfs.nodes), 1)
self.assertEqual(vfs.vpath, "") self.assertEqual(vfs.vpath, "")
self.assertEqual(vfs.realpath, td) self.assertEqual(vfs.realpath, td)
self.assertAxs(vfs.axs.uread, ["*", "k"]) self.assertEqual(vfs.uread, ["*", "k"])
self.assertAxs(vfs.axs.uwrite, ["k"]) self.assertEqual(vfs.uwrite, ["k"])
n = vfs.nodes["a"] n = vfs.nodes["a"]
self.assertEqual(len(vfs.nodes), 1) self.assertEqual(len(vfs.nodes), 1)
self.assertEqual(n.vpath, "a") self.assertEqual(n.vpath, "a")
self.assertEqual(n.realpath, os.path.join(td, "a")) self.assertEqual(n.realpath, os.path.join(td, "a"))
self.assertAxs(n.axs.uread, ["*", "k"]) self.assertEqual(n.uread, ["*", "k"])
self.assertAxs(n.axs.uwrite, ["k"]) self.assertEqual(n.uwrite, ["k"])
n = n.nodes["ac"] n = n.nodes["ac"]
self.assertEqual(len(vfs.nodes), 1) self.assertEqual(len(vfs.nodes), 1)
self.assertEqual(n.vpath, "a/ac") self.assertEqual(n.vpath, "a/ac")
self.assertEqual(n.realpath, os.path.join(td, "a", "ac")) self.assertEqual(n.realpath, os.path.join(td, "a", "ac"))
self.assertAxs(n.axs.uread, ["*", "k"]) self.assertEqual(n.uread, ["*", "k"])
self.assertAxs(n.axs.uwrite, ["k"]) self.assertEqual(n.uwrite, ["k"])
n = n.nodes["acb"] n = n.nodes["acb"]
self.assertEqual(n.nodes, {}) self.assertEqual(n.nodes, {})
self.assertEqual(n.vpath, "a/ac/acb") self.assertEqual(n.vpath, "a/ac/acb")
self.assertEqual(n.realpath, os.path.join(td, "a", "ac", "acb")) self.assertEqual(n.realpath, os.path.join(td, "a", "ac", "acb"))
self.assertAxs(n.axs.uread, ["k"]) self.assertEqual(n.uread, ["k"])
self.assertAxs(n.axs.uwrite, ["*", "k"]) self.assertEqual(n.uwrite, ["*", "k"])
# something funky about the windows path normalization, # something funky about the windows path normalization,
# doesn't really matter but makes the test messy, TODO? # doesn't really matter but makes the test messy, TODO?
@@ -179,24 +173,24 @@ class TestVFS(unittest.TestCase):
# admin-only rootfs with all-read-only subfolder # admin-only rootfs with all-read-only subfolder
vfs = AuthSrv( vfs = AuthSrv(
Cfg(a=["k:k"], v=[".::rw,k", "a:a:r"]), Cfg(a=["k:k"], v=[".::ak", "a:a:r"]),
self.log, self.log,
).vfs ).vfs
self.assertEqual(len(vfs.nodes), 1) self.assertEqual(len(vfs.nodes), 1)
self.assertEqual(vfs.vpath, "") self.assertEqual(vfs.vpath, "")
self.assertEqual(vfs.realpath, td) self.assertEqual(vfs.realpath, td)
self.assertAxs(vfs.axs.uread, ["k"]) self.assertEqual(vfs.uread, ["k"])
self.assertAxs(vfs.axs.uwrite, ["k"]) self.assertEqual(vfs.uwrite, ["k"])
n = vfs.nodes["a"] n = vfs.nodes["a"]
self.assertEqual(len(vfs.nodes), 1) self.assertEqual(len(vfs.nodes), 1)
self.assertEqual(n.vpath, "a") self.assertEqual(n.vpath, "a")
self.assertEqual(n.realpath, os.path.join(td, "a")) self.assertEqual(n.realpath, os.path.join(td, "a"))
self.assertAxs(n.axs.uread, ["*"]) self.assertEqual(n.uread, ["*"])
self.assertAxs(n.axs.uwrite, []) self.assertEqual(n.uwrite, [])
self.assertEqual(vfs.can_access("/", "*"), [False, False, False, False]) self.assertEqual(vfs.can_access("/", "*"), [False, False])
self.assertEqual(vfs.can_access("/", "k"), [True, True, False, False]) self.assertEqual(vfs.can_access("/", "k"), [True, True])
self.assertEqual(vfs.can_access("/a", "*"), [True, False, False, False]) self.assertEqual(vfs.can_access("/a", "*"), [True, False])
self.assertEqual(vfs.can_access("/a", "k"), [True, False, False, False]) self.assertEqual(vfs.can_access("/a", "k"), [True, False])
# breadth-first construction # breadth-first construction
vfs = AuthSrv( vfs = AuthSrv(
@@ -253,26 +247,26 @@ class TestVFS(unittest.TestCase):
./src ./src
/dst /dst
r a r a
rw asd a asd
""" """
).encode("utf-8") ).encode("utf-8")
) )
au = AuthSrv(Cfg(c=[cfg_path]), self.log) au = AuthSrv(Cfg(c=[cfg_path]), self.log)
self.assertEqual(au.acct["a"], "123") self.assertEqual(au.user["a"], "123")
self.assertEqual(au.acct["asd"], "fgh:jkl") self.assertEqual(au.user["asd"], "fgh:jkl")
n = au.vfs n = au.vfs
# root was not defined, so PWD with no access to anyone # root was not defined, so PWD with no access to anyone
self.assertEqual(n.vpath, "") self.assertEqual(n.vpath, "")
self.assertEqual(n.realpath, None) self.assertEqual(n.realpath, None)
self.assertAxs(n.axs.uread, []) self.assertEqual(n.uread, [])
self.assertAxs(n.axs.uwrite, []) self.assertEqual(n.uwrite, [])
self.assertEqual(len(n.nodes), 1) self.assertEqual(len(n.nodes), 1)
n = n.nodes["dst"] n = n.nodes["dst"]
self.assertEqual(n.vpath, "dst") self.assertEqual(n.vpath, "dst")
self.assertEqual(n.realpath, os.path.join(td, "src")) self.assertEqual(n.realpath, os.path.join(td, "src"))
self.assertAxs(n.axs.uread, ["a", "asd"]) self.assertEqual(n.uread, ["a", "asd"])
self.assertAxs(n.axs.uwrite, ["asd"]) self.assertEqual(n.uwrite, ["asd"])
self.assertEqual(len(n.nodes), 0) self.assertEqual(len(n.nodes), 0)
os.unlink(cfg_path) os.unlink(cfg_path)

View File

@@ -31,7 +31,7 @@ if MACOS:
from copyparty.util import Unrecv from copyparty.util import Unrecv
def runcmd(argv): def runcmd(*argv):
p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE) p = sp.Popen(argv, stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = p.communicate() stdout, stderr = p.communicate()
stdout = stdout.decode("utf-8") stdout = stdout.decode("utf-8")
@@ -39,8 +39,8 @@ def runcmd(argv):
return [p.returncode, stdout, stderr] return [p.returncode, stdout, stderr]
def chkcmd(argv): def chkcmd(*argv):
ok, sout, serr = runcmd(argv) ok, sout, serr = runcmd(*argv)
if ok != 0: if ok != 0:
raise Exception(serr) raise Exception(serr)
@@ -60,20 +60,12 @@ def get_ramdisk():
if os.path.exists("/Volumes"): if os.path.exists("/Volumes"):
# hdiutil eject /Volumes/cptd/ # hdiutil eject /Volumes/cptd/
devname, _ = chkcmd("hdiutil attach -nomount ram://131072".split()) devname, _ = chkcmd("hdiutil", "attach", "-nomount", "ram://131072")
devname = devname.strip() devname = devname.strip()
print("devname: [{}]".format(devname)) print("devname: [{}]".format(devname))
for _ in range(10): for _ in range(10):
try: try:
_, _ = chkcmd(["diskutil", "eraseVolume", "HFS+", "cptd", devname]) _, _ = chkcmd("diskutil", "eraseVolume", "HFS+", "cptd", devname)
with open("/Volumes/cptd/.metadata_never_index", "w") as f:
f.write("orz")
try:
shutil.rmtree("/Volumes/cptd/.fseventsd")
except:
pass
return subdir("/Volumes/cptd") return subdir("/Volumes/cptd")
except Exception as ex: except Exception as ex:
print(repr(ex)) print(repr(ex))
@@ -127,13 +119,14 @@ class VHttpConn(object):
self.addr = ("127.0.0.1", "42069") self.addr = ("127.0.0.1", "42069")
self.args = args self.args = args
self.asrv = asrv self.asrv = asrv
self.nid = None self.is_mp = False
self.log_func = log self.log_func = log
self.log_src = "a" self.log_src = "a"
self.lf_url = None self.lf_url = None
self.hsrv = VHttpSrv() self.hsrv = VHttpSrv()
self.nreq = 0 self.nreq = 0
self.nbyte = 0 self.nbyte = 0
self.workload = 0
self.ico = None self.ico = None
self.thumbcli = None self.thumbcli = None
self.t0 = time.time() self.t0 = time.time()