zepter cli

a simple cli to lint and fix recurrent issues in rust workspaces surrounding 'features'. it can be installed with cargo install zepter and is documented here. it is most prominently used in the polkadot-sdk but can be used in any rust repo.

motivation

missing feature configuration can result in unexpected build errors that are very difficult to debug. especially when the dependency tree is deep.
we also kept getting external bug reports that our code would not build in downstream repos. resolving of these issues would take me about an hour; surely this was an unacceptable state.

now there are sadly many ways to mis-configure features in rust. zepter prevents about the three most common ones. it turns out that this was enough to cut the bug report rate down to zero. internal developers thanked me as well that this was a much welcomed change to reduce development time.

architecture

zepter extracts all relevant data from a rust workspace by utilizing cargo metadata. as user input it accepts a set of rules that must not be violated. it then creates a graph (directed-acyclic-graph) and tries to find violations of each rule. if that fails, the program succeeds.

usage

running the cli is enough to detect and fix mostly all issues in our code base. this is possible since polkadot-sdk has a config file that defines the default behaviour.

$ zepter 
[INFO] Running workflow 'default'
[INFO] 1/2 lint propagate-feature
[INFO] 2/2 format features

helpful error messages are provided in the ci through custom config:

Polkadot-SDK uses the Zepter CLI to detect abnormalities in the feature configuration.
It looks like one more more checks failed; please check the console output. You can try to automatically address them by running `zepter`.
Otherwise please ask directly in the Merge Request, GitHub Discussions or on Matrix Chat, thank you.

performance

the dag code is reasonably fast that a run in our codebase (488 crates) with multiple lints takes about 2 seconds. in general there can still be a lot of optimization, especially caching and not re-computing things multiple times. the code was just written very quickly (=optimized for dev speed :P).

the slowest part is the exhaustive path search in the dag, luckily this is embarrassingly parallel and could be optimized. the most memory intense operations are the generation of transitive closures of the dag. in most cases it is enough to build the transitive closure of a subset, which reduces memory consumption.

fellowship dashboard

a small and minimalistic website to monitor the compliance of the polkadot fellows available at fellowship.tasty.limo the code is on github.

architecture

it is written in and deployed to one of my personal servers. every 12 hours the data is fetched from an rpc provider. this is not ideal and should rather use light-clients, but i did not have the time to set that up; 80-20 you know.

the website is compacted into a single page to load within one http get call. this makes it very fast - even with jquery bloat.

the data extracted happens in three steps:

  • loading all members from the polkadot-collectives parachain
  • querying the on-chain identities from the polkadot relay
  • fetching the github bio and checking if their address is mentioned

this is then rendered into a html template and a cache file for hot restarts.

future

i would like to decomission this eventually once the official website. but currently it is not showing the identities or github handles.

personal website

this website uses the zola static page generator and is themed with terminimal.

goals

all the things that my old website were not:

  • simple to maintain
  • aesthetic
  • clear design
  • (mostly) bloat free
  • fast to load

architecture

in zola there are only two types of pages: indices and sections. one example for each would be the list of projects and this post. this means that all content needs to be split up into either of these categories. its quote simple once you understood to mange it.

subweight

subweight is a cli to compare auto-generated rust config files of substrate. there can be thousands of those and manually reviewing them is too tedious. the tool available on crates.io and should work with all substrate blockchains.

prelude

substrate is a framework to write blockchains. it uses "weight" to annotate the estimated worst case resource consumption of an operation. this is crucial since it needs to ensure that all effort is paid for - we cannot "undo" the spent computational effort.

now these weights need to be updated every now and then. this happens automatically but should still be sanity checked by humans. this is where subweight comes into play. it breaks down ugly diffs and makes them graspable.

example

the files that it compares contain of lots of functions and look like this:

/// Storage: `Multisig::Multisigs` (r:1 w:1)
/// Proof: `Multisig::Multisigs` (`max_values`: None, `max_size`: Some(3346), added: 5821, mode: `MaxEncodedLen`)
/// Storage: `System::Account` (r:1 w:1)
/// Proof: `System::Account` (`max_values`: None, `max_size`: Some(128), added: 2603, mode: `MaxEncodedLen`)
/// The range of component `s` is `[2, 100]`.
/// The range of component `z` is `[0, 10000]`.
fn as_multi_complete(s: u32, z: u32, ) -> Weight {
	// Proof Size summary in bytes:
	//  Measured:  `392 + s * (33 ±0)`
	//  Estimated: `6811`
	// Minimum execution time: 45_986_000 picoseconds.
	Weight::from_parts(34_083_317, 0)
		.saturating_add(Weight::from_parts(0, 6811))
		.saturating_add(Weight::from_parts(159_314, 0).saturating_mul(s.into()))
		.saturating_add(Weight::from_parts(1_495, 0).saturating_mul(z.into()))
		.saturating_add(T::DbWeight::get().reads(2))
		.saturating_add(T::DbWeight::get().writes(2))
}

now, having about 1600 of those updated in a single merge request is impossibly to humanly review. subweight can be used in a few different ways to automatically show a comprehensible diff.

for example in merge request 129 it was used to plot a nice overview:

Nice weight diff

there is also a web version available, served by the binary subweight-web.

future

i hope to eventually retire this tool once we have either:

  • a proper intermediate format to represent the formulas
  • automatic metering that can abort execution and still charge fees

the second option is still a bit out, so for now it stays.