# HG changeset patch # User Arne Babenhauserheide # Date 1730623733 -3600 # Sun Nov 03 09:48:53 2024 +0100 # Node ID 21c9b61eb59b3066d04fdef5f77278e445a5867e # Parent dfd41f695042ae7e7e7a0e612729e75329bd530d # Parent 56a2023b87134f6c502cb3fb0229a0435af7c43d merge hacking notes diff --git a/docs/hacking.org b/docs/hacking.org --- a/docs/hacking.org +++ b/docs/hacking.org @@ -94,5 +94,77 @@ - replace trusters trustlist in wotstate +*** Optimization: precise, incremental trust change propagation +- [X] Adding trust only requires recalculating scores of IDs affected by the trust addition +- [X] Removing trust only requires recalculating scores of IDs downstream of the distrusted ID +- [ ] optional speedup: switch to vectors that double size when +they have to be extended (need to track a length independent of +the actual datastructure size) + +* constant cost plan +*** Scalability optimization: constant cost at arbitrary count of users +- [X] Prune stale IDs + - [X] aggregate ranks for all the ownids, global array with + ranks: max rank. Any time pruning is checked, update the max rank + datastructure for all ownids. On getting trust, only the required + ownid is updated. + - [X] Only IDs with rank >= 2 are pruned — direct trustees are + never pruned. + - [X] Pursue a stable state of trust: probabilistically reduce + trust given by non-ownids. Define an upper limit of IDs (in + usual setup 60k because of the datastructure size, when setup + with 32bit indizes, 200x the number of ownids). + - [X] At every trust increase, empty a randomly chosen trustlist + (rank >= 2) with probability =0.02 * (current ID count) / (max ID + count - current ID count)=. This includes the assumption that most + people will have /visible/ trust to about 50 other people (about + half of dunbars number), so we delete an ID as often as we add + one. Interactions by this ID will add trust again. Then update all + scores for all ownids. + - [X] Prune all IDs that have score #f for every ownid. When an id + is pruned, replace it with #f in the known-ids and empty the + trustlist. Interactions with these IDs will add them again as new + known-ids. Score #f means that noone gives trust to you. + - [X] when ~10% of known IDs are #f, reindex all vectors (ids, + trustlists, ranks, scores): re-create known-ids and map old + indexes to new indexes, then use these to re-create the other + vectors from the existing data. +- [ ] Prepare support for non-mastodon sources like RSS + - [ ] Add POST endpoint: IDs with changes → returns next ID to check + - [ ] Split subscriptions by distance to ID as described in + https://www.draketo.de/english/freenet/deterministic-load-decentralized-spam-filter#sec-3-1 + - [ ] Replace for the WoTID the internal ID by the link to each ID, + so other sources can provide trust to mastodon sources +*** Mastodon-support: deduce trust-information +- [X] When connected to the mastodon instance, trust values are deduced from information on that instance and from interaction. +- [X] Implementation details to be fleshed out when the mastodon instance is done. +- [X] Mute-Endpoint (controller): put trust -1; unmute: put trust 0 (visible, not advertised) ⇒ mute + unmute resets trust +- [X] Block-Endpoint: put trust -60 +- [X] Follow-Endpoint: put trust 30; unfollow: trust 0 (stays visible, no longer advertised) +*** Stress testing, optimization, and deployment with a real instance +- [X] Wispwot works in real deployment for a specific mastodon instance. +- [X] The locally running Mastodon instance is filtered and infers trust +- [X] Use persistent trust: write and load to a fixed location. +- [ ] There is documentation how to use wispwot with Mastodon. +- [ ] The wispwot code and documentation is released. The required changes to Mastodon are publicly available. +- mastodon changes: https://github.com/ArneBab/mastodon/pull/1/files +- wispwot-code and documentation: https://hg.sr.ht/~arnebab/wispwot/browse +- REST docs: https://hg.sr.ht/~arnebab/wispwot/browse/HOWTO.org?rev=tip +- deployment: https://hg.sr.ht/~arnebab/wispwot/browse/docs/deployment.org?rev=tip +- datastructures: https://hg.sr.ht/~arnebab/wispwot/browse/docs/hacking.org?rev=tip +* subscriptions + +Subscriptions allow guiding information retrieval and communication +between many different services. + +Changes related to the plan in https://www.draketo.de/english/freenet/deterministic-load-decentralized-spam-filter#sec-3: + +- retrieve new subscriptions at regular intervals, not everytime there + is an update, because clearnet resources can be queried much more + rapidly than those in Hyphanet -> split the endpoint into + =PUT /subscription-updated/ownkey/OWN/otherid/ID= and + =GET /subscriptions/ownid=. + +