Intertwingler MVP

D

A dedicated cache metadata store will be much more efficient in many ways than trying to use RDF.

Other: Index the cache entries using a dedicated key-value store.

Dedicate some time to hunting down and squashing memory leaks and general wasteful memory usage.

Other: There are almost certainly memory leaks in and around Intertwingler and its snarl of dependencies.

Design a compact XSLT syntax and concomitant transpiler.

Other: The XSLT component kind of sucks.; XSLT (any version) would be a million times easier to work with if it was just easier to type.
See Also

Design a mechanism for offloading and/or paginating backlinks.

Has Narrower

Despite being standard, reliable, and fast to execute, and built into every browser, XSLT 1.0 in particular is really clunky to work with.

Other: The XSLT component kind of sucks.; Try messing around with SaxonJS?

Detect what version the metadata layout is and whine at the user to upgrade it on their own time.

Has Broader: Create a mechanism that upgrades old metadata database layouts to the new one.; Create a variant of the metadata driver for both the old and the new version of the database layout.
Has Narrower: Once again, pass in an initialization parameter to explicitly upgrade the store.; Yes. Supply an initialization parameter to ignore the whining.
Other: How does the user assent to upgrading the metadata store?; Should the user be able to silence the whining?; We don't want to take people by surprise by forcing them into upgrading their data store.

doriantaylor.com only needs a basic notion of "private" (only visible to me) and "public".

Other: Implement a rudimentary mechanism for partitioning the graph into "public" and "private".

[DONE] Add an `instance-of` parameter that takes class identifiers for parity with the original resource.

Has Broader: Create an "inventory" resource that is itself a list of resources with the characteristic of being in the range of a given property.; [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.
Other: The original inventory resource in the prototype goes by instance of class rather than range.

[DONE] Clean up slow RDF::Graph methods and push upstream.

Has Broader: Put site-specific data in a named graph that corresponds to the authority, then narrow queries to that named graph.
Other: RDF::Graph slows down operations by two orders of magnitude!

[DONE] Create a Git repository with a prefabricated configuration.

Has Broader: [DONE] Ship an Intertwingler instance that the client can use.
Other: What if the client has trouble?

[DONE] Create a set of "catalogue resources".

Has Broader: [DONE] Port the IBIS tool to Intertwingler.
Has Narrower: Create an "inventory" resource that is itself a list of resources with the characteristic of being in the range of a given property.; Represent windowed slices of a collection as ephemeral entities. Have them refer back to the main collection with the unmanageably many elements.; When somebody tries to access the main collection, redirect them to the first window onto the collection.; [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.; [DONE] Create an index as a top-level entry point to the catalogues.; [DONE] Define a relation that connects the tool to the entry point.
Other: How does the front-end know where the catalogues are?; How will we construct the UI of the RDF-KV forms?; What happens when you have an extreme quantity of entities for a given class and/or in the domain or range of a given predicate?; [RESOLVED] Intertwingler currently does not have the resources to furnish the IBIS tool with fodder for autocompletes.

[DONE] Create an "index" resource that enumerates all the windowed resources.

Other: How does a client determine how to fill out the entire collection?

[DONE] Create an index as a top-level entry point to the catalogues.

Has Broader: [DONE] Create a set of "catalogue resources".
Has Narrower: [DONE] Define a relation that connects the tool to the entry point.
Other: How does the front-end know where the catalogues are?; To populate potential linkages, the tool needs a list of all resources that are in the range of a given property.

[DONE] Create an RDF-KV handler.

Has Broader: [DONE] Ship an Intertwingler instance that the client can use.
Other: How will we construct the UI of the RDF-KV forms?; What about writing to the graph?

[DONE] Create an `asserted` flag that is on by default but can be set to false to eliminate asserted resources from the result set.

Has Broader: [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.
Other: What if you wanted only inferred results with no asserted ones?

[DONE] Create an `inferred` flag that is on by default but can be set to false that will disable inferencing.

Has Broader: [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.
Other: What if we only want resources that match assertions with no inferencing?; What if you wanted only inferred results with no asserted ones?

[DONE] Decouple the IBIS front-end from its back-end (into its own Git[hub] repository).

Has Broader: [DONE] Port the IBIS tool to Intertwingler.
Other: [RESOLVED] The front-end code to the IBIS tool is currently in the back-end repository.
See Also

[DONE] Define a relation that connects the tool to the entry point.

Has Broader

[DONE] Generate an ARM64 Docker image.

Has Broader

[DONE] Install Intertwingler on deuce (which is ostensibly a much, much faster machine than that piddly little VPS on DigitalOcean) and just reverse-proxy it over the VPN.

Has Broader: Get Intertwingler the hell online.
Other: How do we handle authentication and access control?; I don't love having my dev server conscripted into production.; I have fibre and get a 1.7ms ping time to the server, even over ipsec.; It turns out the server at DigitalOcean is actually really slow.; The FastCGI daemon running Intertwingler keeps exiting; there is something not right with it.; We can shell out for proper infrastructure when this thing starts making some money.

[DONE] Make the Intertwingler home directory a Docker volume.

Has Broader: [DONE] Ship Intertwingler as a Docker image.
Other: How do we get instance data in and out?

[DONE] Make the inventory resource respond to an `in-domain-of` parameter for symmetry with `in-range-of`.

Has Broader: Create an "inventory" resource that is itself a list of resources with the characteristic of being in the range of a given property.; [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.
Other: Properties go in the other direction too, so it would be useful to be able to pick subjects of properties as well as objects.

[DONE] Perform RDFS/OWL inferencing against the supplied parameters and result set.

Has Broader: [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.
Other: How do you make sure that you net resources that are subclasses of, or related by subproperties to, the supplied parameters?; What if we only want resources that match assertions with no inferencing?

[DONE] Port the IBIS tool to Intertwingler.

Has Broader: Put makethingsmakesense.com into Intertwingler.; [DONE] Put client extranets into Intertwingler.
Has Narrower: [DONE] Create a set of "catalogue resources".; [DONE] Decouple the IBIS front-end from its back-end (into its own Git[hub] repository).
Other: Client wants to use the IBIS tool (Sense Atlas) in everyday operations.; We would actually be able to represent real sequences of actions instead of just suggestions about what to do.; We would actually be able to unblock development on the IBIS tool that has been sitting idle for over a decade.; [RESOLVED] Intertwingler currently does not have the resources to furnish the IBIS tool with fodder for autocompletes.; [RESOLVED] The front-end code to the IBIS tool is currently in the back-end repository.

[DONE] Put client extranets into Intertwingler.

Has Broader: Put methodandstructure.com into Intertwingler.
Has Narrower: [DONE] Port the IBIS tool to Intertwingler.

[DONE] Put Sense Atlas on its own website, senseatlas.net.

Other: Intertwingler currently lacks any internal caching, so I don't trust it standing up to even the baseline thrash of the open internet.; Where should Sense Atlas go?

[DONE] Put Sense Atlas online so the client can observe its development.

Has Broader: Rough out the new classes and try loading them with instance data; take notes for what can be improved.
Has Narrower: Get Intertwingler the hell online.; Leave Forget Passwords on the DO server, and hack both it and Intertwingler, respectively, to transmit and receive a JWT.; [DONE] Upgrade DigitalOcean server to handle the additional load introduced by Intertwingler.
Other: How do we handle authentication and access control?; Where should Sense Atlas go?

[DONE] Put the `in-range-of` property into parameter.

Has Broader: Create an "inventory" resource that is itself a list of resources with the characteristic of being in the range of a given property.; [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.
Other: Properties go in the other direction too, so it would be useful to be able to pick subjects of properties as well as objects.; The original inventory resource in the prototype goes by instance of class rather than range.; There are all manners of properties, so hard-coding a single one is silly.

[DONE] Ship an Intertwingler instance that the client can use.

Has Narrower: [DONE] Create a Git repository with a prefabricated configuration.; [DONE] Create an RDF-KV handler.; [DONE] Ship Intertwingler as a Docker image.; [DONE] Write a getting-started tutorial.
Other: Client wants to use Intertwingler as a back-end.; It turns out to that the default (AMD64) build is unusably slow in the client's development environment (Darwin/ARM64).; What about writing to the graph?; What if the client has trouble?

[DONE] Ship Intertwingler as a Docker image.

Has Broader: [DONE] Ship an Intertwingler instance that the client can use.
Has Narrower: [DONE] Generate an ARM64 Docker image.; [DONE] Make the Intertwingler home directory a Docker volume.
Other: About half a dozen dependencies have patches that have yet to be merged upstream.; Client is not a Ruby developer.; How do we get instance data in and out?; It turns out to that the default (AMD64) build is unusably slow in the client's development environment (Darwin/ARM64).; Shipping Intertwingler as a Docker image will afford putting off the open-source diplomacy of the upstream patches.

[DONE] Upgrade DigitalOcean server to handle the additional load introduced by Intertwingler.

Has Broader: Install Intertwingler on DigitalOcean VPS as a FastCGI daemon.; [DONE] Put Sense Atlas online so the client can observe its development.
Other: Intertwingler has been observed consuming up to a gigabyte of resident-set RAM during development.; It turns out the server at DigitalOcean is actually really slow.

[DONE] Write a getting-started tutorial.

Has Broader: [DONE] Ship an Intertwingler instance that the client can use.
Other: What if the client has trouble?

Embedding Forget Passwords into Intertwingler is going to severely complicate (and compromise) the configuration for both systems, as well as the installation as a whole.

Other: Split Forget Passwords into app and middleware and wrap the latter around Intertwingler (on deuce).

H

Have the resolver wrap its repository in an RDF::Graph.

Other

Having to pass the graph name into every query is going to be tedious and error-prone.

Other: Have the resolver wrap its repository in an RDF::Graph.; Put site-specific data in a named graph that corresponds to the authority, then narrow queries to that named graph.

Hooking cache evictions to accessors is a simple solution.

Other: Put a hook on the store's accessors to evict the expired cache entries.

How about pay money for proper infrastructure?

Other: It turns out the server at DigitalOcean is actually really slow.; We can shell out for proper infrastructure when this thing starts making some money.

How are the parameters reconciled if you have multiples at once?

Other: [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.; [ADOPTED] Treat the `instance-of`, `in-range-of`, `in-domain-of` parameters as a union, that is, select all resources that match any of them.

How are we going to handle cache entity bodies?

Other: Implement caching of generated representations against Store::Digest.; Implement internal caching.

How are we going to represent the metadata for the cache entries?

Has Narrower: How do we handle private cache entries?
Other: Create an ontology based on PROV-O and the HTTP message ontology that can represent the appropriate metadata.; Implement internal caching.; Index the cache entries using a dedicated key-value store.; Put the cache metadata into the RDF store.

How do we actually go about evicting expired cache entries?

Other: Create a background worker thread that polls regularly for expired cache entries.; Extend Store::Digest to include a way to indicate that an entry ought to be treated as cache.; Put a hook on the store's accessors to evict the expired cache entries.

How do we ensure that we return quickly and not hang up on the cache eviction process?

Other: Put a hook on the store's accessors to evict the expired cache entries.; Run the cache eviction process in another thread.

How do we get instance data in and out?

Other: [DONE] Make the Intertwingler home directory a Docker volume.; [DONE] Ship Intertwingler as a Docker image.

How do we handle authentication and access control?

Other: Leave Forget Passwords on the DO server, and hack both it and Intertwingler, respectively, to transmit and receive a JWT.; Split Forget Passwords into app and middleware and wrap the latter around Intertwingler (on deuce).; [DONE] Install Intertwingler on deuce (which is ostensibly a much, much faster machine than that piddly little VPS on DigitalOcean) and just reverse-proxy it over the VPN.; [DONE] Put Sense Atlas online so the client can observe its development.

How do we handle private cache entries?

Has Broader: How are we going to represent the metadata for the cache entries?
Other: Associate private cache entities with the principal that requested them.

How do we keep drafts and other works in progress from being exposed to the public?

Other: Implement a rudimentary mechanism for partitioning the graph into "public" and "private".; Put doriantaylor.com into Intertwingler.

How do we resolve RFC6920 content addresses to their HTTP(S) counterparts?

Has Broader: There are plenty of other durable URI schemes besides `urn:uuid:` and `ni:` (such as `ark:` or `doi:`).
Other: Overhaul the resolver so that it has pluggable transformations between durable identifiers and routable addresses.; We want Intertwingler to have a unified interface to opaque blob storage, whether a given blob is intended to be persisted or not.
See Also

How do you get the users into the system?

Other

How do you make sure that you net resources that are subclasses of, or related by subproperties to, the supplied parameters?

Other: [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.; [DONE] Perform RDFS/OWL inferencing against the supplied parameters and result set.

How do you safely store API credentials?

Other: Create an API bridge that will connect the two systems.

How does a client determine how to fill out the entire collection?

Other: Attach "next" and "previous" relations to the window resources, when applicable.; Represent windowed slices of a collection as ephemeral entities. Have them refer back to the main collection with the unmanageably many elements.; [DONE] Create an "index" resource that enumerates all the windowed resources.

How does the front-end know where the catalogues are?

Other: [DONE] Create a set of "catalogue resources".; [DONE] Create an index as a top-level entry point to the catalogues.; [DONE] Define a relation that connects the tool to the entry point.

How does the user assent to upgrading the metadata store?

Other: Detect what version the metadata layout is and whine at the user to upgrade it on their own time.; Once again, pass in an initialization parameter to explicitly upgrade the store.

How is the inventory going to be related to the index?

Other: [ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.

How will we construct the UI of the RDF-KV forms?

Has Narrower: [RESOLVED] Intertwingler currently does not have the resources to furnish the IBIS tool with fodder for autocompletes.
Other: [DONE] Create a set of "catalogue resources".; [DONE] Create an RDF-KV handler.

A separate RDF store per site will make a big mess with Intertwingler's configuration data.

Other: Arrange for each site under management to have its own RDF store.

Shipping Intertwingler as a Docker image will afford putting off the open-source diplomacy of the upstream patches.

Other: [DONE] Ship Intertwingler as a Docker image.

Some of the inputs (e.g. via document() function) need to be partially transformed but not go through the XSLT filter.

Other: Create a server-side XSLT transform.

The way the extranets are configured, it would be a big pain to carve out an exception for just one of them.

Other: For now, just configure Intertwingler to power this one client's extranet.

We don't really have time to write migration code and we're not sure how many there are out there in the wild.

Other: Create a mechanism that upgrades old metadata database layouts to the new one.

We would actually be able to represent real sequences of actions instead of just suggestions about what to do.

Other: [DONE] Port the IBIS tool to Intertwingler.

We would actually be able to unblock development on the IBIS tool that has been sitting idle for over a decade.

Other: [DONE] Port the IBIS tool to Intertwingler.

A

About half a dozen dependencies have patches that have yet to be merged upstream.

Adding a cache field to Store::Digest will require changing the metadata record, which in turn will create an incompatibility.

Alias privatealpha.com to methodandstructure.com.

Arrange for each site under management to have its own RDF store.

Associate private cache entities with the principal that requested them.

Associate the cache entity with a null principal to indicate that it is actually public.

ATProto speaks XRPC (over WebSockets) while Intertwingler is very heavily REST.

Attach "next" and "previous" relations to the window resources, when applicable.

The address space is already occupied by the old statically-generated sites and the old App::IBIS prototype.

[ADOPTED] Create a generic inventory resource that can take any combination of filtering parameters.

[ADOPTED] Treat the `instance-of`, `in-range-of`, `in-domain-of` parameters as a union, that is, select all resources that match any of them.

B

Bust out the profiler and fix whatever's causing RDF::Graph to run (INSANELY) slow.

C

Can the resource identified by the hash URI be the cache entry?

Change the LMDB driver on Store::Digest so the main entry table uses integer keys rather than the much larger "primary" hash.

Changing the metadata layout in Store::Digest will make it incompatible with already-initialized stores.

Client is not a Ruby developer.

Client wants to map IBIS issues to GitHub issues.

Client wants to use Intertwingler as a back-end.

Client wants to use the IBIS tool (Sense Atlas) in everyday operations.

Create a background worker thread that polls regularly for expired cache entries.

Create a configurable "pecking order" of URI schemes so the precedence is unambiguous.

Create a mechanism that upgrades old metadata database layouts to the new one.

Create a server-side XSLT transform.

Create a single source of truth within an organization for URIs that get exposed to its edge.

Create a variant of the metadata driver for both the old and the new version of the database layout.

Create a wrapper for the Store::Digest entry objects so they can cosplay as IO objects.

Create an "inventory" resource that is itself a list of resources with the characteristic of being in the range of a given property.

Create an adapter that will map between ordinary HTTP resources and WebSockets.

Create an API bridge that will connect the two systems.

Create an ontology based on PROV-O and the HTTP message ontology that can represent the appropriate metadata.

Create an `etime` index for cache expiry; only put actual tombstones in the `dtime` index, and only put cache entries in `etime`.

D

A dedicated cache metadata store will be much more efficient in many ways than trying to use RDF.

Dedicate some time to hunting down and squashing memory leaks and general wasteful memory usage.

Design a compact XSLT syntax and concomitant transpiler.

Design a mechanism for offloading and/or paginating backlinks.

Despite being standard, reliable, and fast to execute, and built into every browser, XSLT 1.0 in particular is really clunky to work with.

Detect what version the metadata layout is and whine at the user to upgrade it on their own time.

Do not let the file system dictate the content of URLs exposed to the edge.

Do we actually want to keep this domain alive?

Do we actually want/need really busy, chatty cache metadata in the RDF store?

doriantaylor.com only needs a basic notion of "private" (only visible to me) and "public".

[DONE] Add an `instance-of` parameter that takes class identifiers for parity with the original resource.

[DONE] Clean up slow RDF::Graph methods and push upstream.

[DONE] Create a Git repository with a prefabricated configuration.

[DONE] Create a set of "catalogue resources".

[DONE] Create an "index" resource that enumerates all the windowed resources.

[DONE] Create an index as a top-level entry point to the catalogues.

[DONE] Create an RDF-KV handler.

[DONE] Create an `asserted` flag that is on by default but can be set to false to eliminate asserted resources from the result set.

[DONE] Create an `inferred` flag that is on by default but can be set to false that will disable inferencing.

[DONE] Decouple the IBIS front-end from its back-end (into its own Git[hub] repository).

[DONE] Define a relation that connects the tool to the entry point.

[DONE] Generate an ARM64 Docker image.

[DONE] Install Intertwingler on deuce (which is ostensibly a much, much faster machine than that piddly little VPS on DigitalOcean) and just reverse-proxy it over the VPN.

[DONE] Make the Intertwingler home directory a Docker volume.

[DONE] Make the inventory resource respond to an `in-domain-of` parameter for symmetry with `in-range-of`.

[DONE] Perform RDFS/OWL inferencing against the supplied parameters and result set.

[DONE] Port the IBIS tool to Intertwingler.

[DONE] Put client extranets into Intertwingler.

[DONE] Put Sense Atlas on its own website, senseatlas.net.

[DONE] Put Sense Atlas online so the client can observe its development.

[DONE] Put the `in-range-of` property into parameter.

[DONE] Ship an Intertwingler instance that the client can use.

[DONE] Ship Intertwingler as a Docker image.

[DONE] Upgrade DigitalOcean server to handle the additional load introduced by Intertwingler.

[DONE] Write a getting-started tutorial.

E

E-mail addresses are still going out in the clear.

Embedding Forget Passwords into Intertwingler is going to severely complicate (and compromise) the configuration for both systems, as well as the installation as a whole.

Even ordinary page GETs when the graph state has not changed can be slow, particularly the first time a user loads a generated resource.

Extend Store::Digest to include a way to indicate that an entry ought to be treated as cache.

F

File a bug upstream to get proper support for FastCGI in Rackup.

Find out if SHACL also uses grapheme clusters.

Finish implementing addressable transforms.

Fish the FastCGI driver out of old Rack code and install it under /var/lib/intertwingler.