Commit Graph

457 Commits

Author SHA1 Message Date
ed
36f2c446af opengraph stuff:
* template-based title formatting
* picture embeds are no longer ant-sized
* `--og-color` sets accent color; default #333
* `--og-s-title` forces default title, ignoring e2t
* add a music indicator to song titles because discord doesn't
2024-05-03 00:11:40 +00:00
ed
69517e4624 add general-purpose query-string parcelling;
currently only being used to workaround discord discarding
query strings in opengraph tags, but i'm sure there will be
plenty more wonderful usecases for this atrocity
2024-05-02 22:49:27 +00:00
ed
ea270ab9f2 add og / opengraph / discord embeds 2024-05-01 23:40:56 +00:00
ed
b6cf2d3089 --html-head can take a filepath and/or jinja2 2024-05-01 20:24:18 +00:00
ed
27485a4cb1 add pyz builder 2024-04-24 23:45:01 +00:00
ed
f6e693f0f5 reevaluate support for sparse files periodically
if a given filesystem were to disappear (e.g. removable storage)
followed by another filesystem appearing at the same location,
this would not get noticed by up2k in a timely manner

fix this by discarding the mtab cache after `--mtab-age` seconds and
rebuild it from scratch, unless the previous values are definitely
correct (as indicated by identical output from `/bin/mount`)

probably reduces windows performance by an acceptable amount
2024-04-24 21:18:26 +00:00
ed
4bb0e6e75a pipe: windows: make it safe with aggressive flushing 2024-04-20 22:15:08 +00:00
ed
f31ac90932 less confusing help-text for --re-dhash 2024-04-20 16:42:56 +00:00
ed
c06126cc9d pipe: add volflag to disable 2024-04-19 23:54:23 +00:00
ed
bf5850785f add opt-out from storing uploader IPs 2024-04-18 17:16:00 +00:00
ed
c8e3ed3aae retry failed renames on windows
theoretical issue which nobody has ran into yet,
probably because nobody uses this on windows
2024-04-12 20:38:30 +00:00
ed
7b94e4edf3 configurable basic-auth preference;
adds options `--bauth-last` to lower the preference for
taking the basic-auth password in case of conflict,
and `--no-bauth` to entirely disable basic-authentication

if a client is providing multiple passwords, for example when
"logged in" with one password (the `cppwd` cookie) and switching
to another account by also sending a PW header/url-param, then
the default evaluation order to determine which password to use is:

url-param `pw`, header `pw`, basic-auth header, cookie (cppwd/cppws)

so if a client supplies a basic-auth header, it will ignore the cookie
and use the basic-auth password instead, which usually makes sense

but this can become a problem if you have other webservers running
on the same domain which also support basic-authentication

--bauth-last is a good choice for cooperating with such services, as
--no-bauth currently breaks support for the android app...
2024-04-11 20:15:49 +00:00
ed
5ab54763c6 remove pyoxidizer (unmaintained)
partially reverts e430b2567a

the remaining stuff might be useful for other cpython alternatives
2024-04-05 17:51:26 +00:00
ed
a080759a03 add transcoding to mp3
because CU's car stereo can't do opus...

incidentally adds support for playing any audio format in ie11
2024-03-29 16:36:56 +00:00
ed
20870fda79 Merge branch 'dirkeys' into hovudstraum 2024-03-25 10:34:08 +00:00
ed
c6acd3a904 add option --s-rd-sz (socket read size):
counterpart of `--s-wr-sz` which existed already

the default (256 KiB) appears optimal in the most popular scenario
(linux host with storage on local physical disk, usually NVMe)

was previously 32 KiB, so large uploads should now use 17% less CPU

also adds sanchecks for values of `--iobuf`, `--s-rd-sz`, `--s-wr-sz`

also adds file-overwrite feature for multipart posts
2024-03-23 16:35:14 +00:00
ed
2b24c50eb7 add option --iobuf (file r/w buffersize):
the default (256 KiB) appears optimal in the most popular scenario
(linux host with storage on local physical disk, usually NVMe)

was previously a mix of 64 and 512 KiB;
now the same value is enforced everywhere

download-as-tar is now 20% faster with the default value
2024-03-23 16:17:40 +00:00
ed
51d31588e6 parse xff before deciding to reject a connection
this commit partially fixes the following issue:
if a client manages to escape real-ip detection, copyparty will
try to ban the reverse-proxy instead, effectively banning all clients

this can happen if the configuration says to obtain client real-ip
from a cloudflare header, but the server is not configured to reject
connections from non-cloudflare IPs, so a scanner will eventually
hit the server IP with malicious-looking requests and trigger a ban

copyparty will now continue to process requests from banned IPs until
the header has been parsed and the real-ip has been obtained (or not),
causing an increased server load from malicious clients

assuming the `--xff-src` and `--xff-hdr` config is correct,
this issue should no longer be hitting innocent clients

the old behavior of immediately rejecting a banned IP address
can be re-enabled with the new option `--early-ban`
2024-03-17 02:36:03 +00:00
ed
66abf17bae black 2024-03-14 18:37:05 +00:00
ed
b377791be7 support cidr notation for --xff-src, --ipa, --*-ipa
the old `10.88.` syntax is still supported,
translating to `10.88.0.0/16`

also fix `--tftp-ipa` when optimizations are enabled
2024-03-14 19:07:35 +01:00
ed
d71f844b43 IdP: add safeguard --idp-h-key and also require --xff-src 2024-03-12 22:57:47 +00:00
ed
e8b7f65f82 IdP: parallel user init + rename idp-h-sep to idp-gsep
`--idp-h-sep` is still supported and will map to its new name
2024-03-12 21:21:53 +00:00
ed
f193f398c1 Merge branch 'hovudstraum' into idp 2024-03-12 17:31:27 +00:00
ed
3f05b6655c add UI to abort an unfinished upload; suggested in #77
to abort an upload, refresh the page and access the unpost tab,
which now includes unfinished uploads (sorted before completed ones)

can be configured through u2abort (global or volflag);
by default it requires both the IP and account to match

https://a.ocv.me/pub/g/nerd-stuff/2024-0310-stoltzekleiven.jpg
2024-03-11 01:32:02 +01:00
ed
1c011ff0bb hide k304 config from controlpanel by default;
as this option is very rarely useful, add global-option `--k304` to
unhide the button and/or set it default-enabled

the toggle will still appear when the feature was previously enabled by
a client, and the feature is still default-enabled for all IE clients
2024-03-09 17:50:24 +00:00
ed
d744f3ff8f improve smoketests, warnings and error-messages:
* docker: warn if there are config-files in ~/.config/copyparty
   because somebody copied their config into
   /cfg/copyparty instead of /cfg as intended

* docker: warn if there are no config-files in an included directory

* make misconfigured reverse-proxies more obvious
  * explain cors rejections in server log
  * indicate cors rejection in error toast
2024-03-07 19:47:38 +00:00
ed
1b52ef1f8a Merge branch 'hovudstraum' into idp 2024-02-23 22:25:48 +00:00
ed
33f41f3e61 add hi-res thumbs (togglebtn/servercfg) 2024-02-18 13:04:22 +00:00
ed
0504b010a1 tftp: support ipv6 and utf-8 filenames + ...
* fix winexe
* missing newline after dirlist
* optimizations
2024-02-17 21:31:58 +00:00
ed
acbb8267e1 tftp: add directory listing 2024-02-10 23:50:17 +00:00
ed
8796c09f56 add --tftp-pr to specify portrange instead of ephemerals 2024-02-10 21:45:57 +00:00
ed
d636316a19 add tftp server 2024-02-10 18:37:21 +00:00
ed
a96d9ac6cb idp: users can be in multiple groups 2024-02-08 20:25:32 +00:00
ed
35165f8472 Merge branch 'hovudstraum' into idp 2024-02-03 19:14:49 +00:00
ed
c354a38b4c up2k: warn about browser cap on num connections 2024-02-02 23:46:00 +00:00
ed
caf7e93f5e IdP (#62): add groups + dynamic vols (non-persistent)
features which should be good to go:
* user groups
* assigning permissions by group
* dynamically created volumes based on username/groupname
* rebuild vfs when new users/groups appear

but several important features still pending;
* detect dangerous configurations
   * dynamic vol below readable path
* remember volumes created during previous runs
   * helps prevent unintended access
   * correct filesystem-scan on startup
2024-01-30 19:13:42 +01:00
ed
14bccbe45f backports from IdP branch:
* allow mounting `/` (the entire filesystem) as a volume
  * not that you should (really, you shouldn't)
* improve `-v` helptext
* change IdP group symbol to @ because % is used for file inclusion
  * not technically necessary but is less confusing in docs
2024-01-25 21:39:30 +00:00
ed
b9d0c8536b avoid sendfile bugs on 32bit machines:
https://github.com/python/cpython/issues/114077
2024-01-17 20:56:44 +00:00
ed
3313503ea5 retry deleting busy files on windows:
some clients (clonezilla-webdav) rapidly create and delete files;
this fails if copyparty is still hashing the file (usually the case)

and the same thing can probably happen due to antivirus etc

add global-option --rm-retry (volflag rm_retry) specifying
for how long (and how quickly) to keep retrying the deletion

default: retry for 5sec on windows, 0sec (disabled) on everything else
because this is only a problem on windows
2024-01-17 20:27:53 +00:00
ed
95a599961e add RAM usage tracking to thumbnailer;
prevents server OOM from high RAM usage by FFmpeg when generating
spectrograms and waveforms: https://trac.ffmpeg.org/ticket/10797
2024-01-14 04:15:09 +00:00
ed
d0eb014c38 improve applefilters + add missing newline in curl 404
* webdav: extend applesan regex with more stuff to exclude
* on macos, set applesan as default `--no-idx` to avoid indexing them
   (they didn't show up in search since they're dotfiles, but still)
2024-01-12 00:13:35 +01:00
ed
86419b8f47 suboptimizations and some future safeguards 2024-01-10 23:20:42 +01:00
ed
987caec15d v1.9.28 2023-12-31 18:49:42 +00:00
ed
ab40ff5051 add permission "A" (alias of "rwmda."); closes #70 2023-12-31 18:20:24 +00:00
ed
5b26ab0096 add option to specify default num parallel uploads 2023-12-28 01:41:17 +01:00
ed
59688bc8d7 * rename hdr-au-usr to idp-h-usr
* ensure lowercase idp-h-*, xff-hdr
* more macos support in tooling
2023-12-24 13:46:12 +01:00
ed
10bc2d9205 unsuccessful attempt at dirkeys (#64) 2023-12-17 22:30:22 +00:00
ed
0c50ea1757 list dotfiles only for specific volumes or users (#66):
* permission `.` grants dotfile visibility if user has `r` too
* `-ed` will grant dotfiles to all `r` accounts (same as before)
* volflag `dots` likewise

also drops compatibility for pre-0.12.0 `-v` syntax
(`-v .::red` will no longer translate to `-v .::r,ed`)
2023-12-16 15:38:48 +00:00
ed
c057c5e8e8 extend --th-covers with dotfiles; closes #67 2023-12-14 10:53:15 +00:00
ed
cba2e10d29 cleanup 2023-12-14 10:47:52 +00:00
ed
91f010290c improve --help descriptions 2023-12-03 02:35:38 +00:00
ed
626b5770a5 add --ftp-ipa 2023-11-30 23:36:46 +00:00
ed
00812cb1da new option --ipa; client IP allowlist:
connections from outside the specified list of IP prefixes are rejected
(docker-friendly alternative to -i 127.0.0.1)

also mkdir any missing folders when logging to file
2023-11-30 20:45:43 +00:00
ed
5e932a9504 hilight metavars in help text 2023-11-30 18:19:34 +00:00
ed
ccab44daf2 initial support for identity providers (#62):
add argument --hdr-au-usr which specifies a HTTP header to read
usernames from; entirely bypasses copyparty's password checks
for http/https clients (ftp/smb are unaffected)

users must exist in the copyparty config, passwords can be whatever

just the first step but already a bit useful on its own,
more to come in a few months
2023-11-30 18:18:47 +00:00
ed
8c52b88767 make linters happier 2023-11-30 17:33:07 +00:00
ed
c9fd26255b support environment variables mostly everywhere,
useful for docker/systemd stuff

also makes logfiles flush to disk per line by default;
can be disabled for a small performance gain with --no-logflush
2023-11-30 10:22:52 +00:00
ed
ef33ce94cd filter shadowed files from search results (#61),
also adds optimization to stop opening cursors
when max results has already been hit
2023-11-19 11:04:36 +00:00
ed
4b720f4150 add more prometheus metrics; breaking changes:
* cpp_uptime is now a gauge
* cpp_bans is now cpp_active_bans (and also a gauge)

and other related fixes:
* stop emitting invalid cpp_disk_size/free for offline volumes
* support overriding the spec-mandatory mimetype with ?mime=foo
2023-11-04 20:32:34 +00:00
ed
95566e8388 cosmetics:
* fix toast/tooltip colors on splashpage
* properly warn if --ah-cli or --ah-gen is used without --ah-alg
* support ^D during --ah-cli
* improve flavor texts
2023-11-03 16:52:43 +00:00
ed
1f75314463 placeholder expansion in readme and logues; closes #56
also fixes the "scan" volflag which broke in v1.9.14
2023-10-24 16:37:32 +00:00
ed
fc658e5b9e utcfromtimestamp was deprecated and nobody told me,
not even the deprecationwarning that got silently generated burning
20~30% of all CPU-time without actually displaying it anywhere, nice

python 3.12.0 is now only 5% slower than 3.11.6

also fixes some other, less-performance-fatal deprecations
2023-10-20 23:41:58 +00:00
ed
e1d78d8b23 increase timeout of unfinished uploads from 6 to 24 hours
plus make it configurable
2023-10-20 18:31:28 +00:00
ed
801da8079b only 404-ban accounts with permission [gGh]:
never bonk anyone with read-access (able to see directory-listing)
or write-only (not able to retrieve any files at all) due to
either --ban-404 or --ban-url

fixes accidental ban when webdav-uploading files which
match any of the --ban-url patterns (#55)

also default-enables --ban-404 since it is now generally safe
(even when up2k is in turbo mode), plus make turbo smart enough to
disengage when necessary
2023-10-18 22:14:09 +00:00
ed
4b5a0787ab option to show upload timestamps in directory listing;
enable with -mte +.ip_at
or volflag mte=+.ip_at

worst-case performance impact: 18%
2023-10-17 17:51:27 +00:00
ed
9ca8154651 prefer the new TTF in pillow 10.1 + pyinstaller 6.1 fixes 2023-10-15 18:47:34 +00:00
ed
2df76eb6e1 client decides if thumbnails should be cropped or not
this carries some intentional side-effects; each thumbnail format will
now be stored in its own subfolder under .hist/th/ making cleanup more
effective (jpeg and webm are dropped separately)
2023-10-15 10:21:25 +00:00
ed
dc2b67f155 ui-button to use upload-time instead of local last-modified 2023-10-15 08:46:23 +00:00
ed
9f32e9e11d set default sort order; --sort or volflag "sort" 2023-10-14 22:17:37 +00:00
ed
8f2d502d4d configurable printing of failed login attempts 2023-10-08 20:41:02 +00:00
ed
753e3cfbaf revert 68c6794d (v1.6.2) and fix it better:
moving deduplicated files between volumes could drop some links
2023-10-07 22:25:44 +00:00
ed
0dc3c23b42 add alternative filekey generator; closes #52 2023-10-06 13:41:22 +00:00
ed
fc9b729fc2 fix #51:
* handle unexpected localstorage values
* handle unsupported --lang values
2023-09-30 22:54:21 +00:00
ed
163e3fce46 improve reverse-proxy support when containerized:
the x-forwarded-for header would get rejected since the reverse-proxy
is not asking from 127.0.0.1 or ::1, so make this allowlist configurable
2023-09-22 22:39:20 +00:00
ed
e76a50cb9d add indexer benchmark + bump default num cores from 4 to 5
and make the mtag deps build better on fedora
2023-09-22 20:40:52 +00:00
ed
c47047c30d configurable real-ip header from reverse proxy 2023-09-20 21:56:39 +00:00
ed
18c763ac08 smb: upgrade to impacket 0.11, full user account support,
permissions are now per-account instead of coalescing

also stops windows from freaking out if there's an offline volume
2023-09-09 12:46:37 +00:00
ed
5a968f9e47 add permission 'h': folders redirect to index.html;
safest way to make copyparty like a general-purpose webserver where
index.html is returned as expected yet directory listing is entirely
disabled / unavailable
2023-09-07 23:30:01 +00:00
ed
767696185b add ?tar=gz, ?tar=bz2, ?tar=xz with optional level;
defaults are ?tar=gz:3, ?tar=bz2:9, ?tar=xz:1
2023-09-01 23:44:10 +00:00
ed
50e01d6904 add more autoban triggers:
* --ban-url: URLs which 404 and also match --sus-urls (bot-scan)
* --ban-403: trying to access volumes that dont exist or require auth
* --ban-422: invalid POST messages, fuzzing and such
* --nonsus-urls: regex of 404s which  shouldn't trigger --ban-404

in may situations it makes sense to handle this logic inside copyparty,
since stuff like cloudflare and running copyparty on another physical
box than the nginx frontend is on becomes fairly clunky
2023-08-26 13:52:24 +00:00
ed
9b46291a20 add option to force-disable turbo,
making it safer to enable --ban-404
(u2c can still get banned inadvertently)
2023-08-26 13:19:38 +00:00
ed
db0abbfdda typo 2023-08-21 00:05:39 +00:00
ed
cafe53c055 v1.9.0 2023-08-20 22:02:40 +00:00
ed
fc0405c8f3 add prometheus metrics; closes #49 2023-08-20 17:58:06 +00:00
ed
8341041857 mdns: option to ignore spec to avoid issues on
networks where clients have multiple IPs of which some are subnets that
the copyparty server is not
2023-08-19 21:45:26 +00:00
ed
1b7634932d tar/zip-download: add opus transcoding filter 2023-08-19 19:40:46 +00:00
ed
4f80e44ff7 option to exactly specify browser title prefix 2023-08-15 03:17:01 +02:00
ed
bee26e853b show server hostname in html titles:
* --doctitle defines most titles, prefixed with "--name: " by default
* the file browser is only prefixed with the --name itself
* --nth ("no-title-hostname") removes it
* also removed by --nih ("no-info-hostname")
2023-08-14 23:50:13 +02:00
ed
04a1f7040e adjustable timestamp resolution in log messages 2023-08-14 17:22:22 +02:00
ed
e17bf8f325 require the new admin permission for the admin-panel 2023-07-15 18:39:41 +00:00
ed
22fc4bb938 add event-hook for banning users 2023-07-13 22:29:32 +00:00
ed
551d99b71b add permission "a" to show uploader IPs (#45) 2023-07-12 21:36:55 +00:00
ed
b54b7213a7 more thumbnailer configs available as volflags:
--th-convt = convt
--th-no-crop = nocrop
--th-size = thsize
2023-07-11 22:15:37 +00:00
ed
a10cad54fc v1.8.1 2023-07-07 22:20:01 +00:00
ed
5d8cb34885 404/403 can be handled with plugins 2023-07-07 21:33:40 +00:00
ed
8d248333e8 dont disable quickedit when hashing passwords interactively 2023-07-07 18:29:30 +00:00
ed
a0c1239246 v1.8.0 2023-06-26 00:05:12 +00:00
ed
baaf2eb24d include mdns names in tls cert 2023-06-25 22:06:35 +00:00
ed
e197895c10 support hashed passwords; closes #39 2023-06-25 21:50:33 +00:00
ed
8b0cf2c982 volflags to limit volume size / num files; closes #40 2023-06-19 00:42:45 +00:00