(unofficial) polkadot status page

the fact that polkadot does not have a status page is slightly worrying to me. all service providers to have a public communication channel to report on incidents and outages in real time.

status pages seem to do this job quite well in the past and we should have one, too.

design

the current prototype of dotpong sends a remark transaction to the rococo relay chain and rococo asset-hub about every 60 seconds. it then reports the inclusion time back to the status page via webhook.

this is a very primitive design and not a good benchmark, but i am aiming at quantifying end-user experience - not performance.

future

karim jedda from parity is currently prototyping in the same direction. hopefully, we can join our efforts and create a proper status page for polkadot.

polkadot tooling collective

this document proposes to the polkadot tokenholders the inception of a new system collective: the polkadot tooling collective (potoc).

after a formation period to establish members and display effectiveness, the collective will request funds from the main treasury to support a salary structure and sub-treasury. these ideas are laid out in the phases section.

context

the polkadot core fellowship is currently the only system collective for developers. this makes it the go-to spot for new developers who want to join polkadot - regardless of the field in which they want to work.

the scope of the core fellowship is rather narrow, focusing primarily on implementations of the polkadot host and runtime code. but many developers are working on tools related to making polkadot successful, both the maintenance and development of the core protocol and the libraries and abstractions for dapp developers. these developers often find that their contributions are not recognized within the mandate of the core fellowship.

but the system can host many collectives. the core fellowship should be just one of many collectives that serve polkadot in different ways. each has its own set of members, scope, and expertise levels.

having a narrow scope is critical for a collective to ensure that it can fulfill its mission. widening the scope leads to a softening of its direction and social cohesion and possibly a loss of vision.

the polkadot tooling collective will serve polkadot by recognizing and retaining developers who make significant contributions to tools and libraries that make polkadot easier to maintain, use, and integrate.

the mission

the polkadot tooling collective acts in the best interest of polkadot. it tries to have a positive impact on developer success in the polkadot ecosystem and intends to do this by working towards two main goals:

• to maintain developer tools to build polkadot itself (core tooling)
• to maintain developer tools to build on polkadot (dapp tooling)

since these definitions are not sufficiently precise, an exhaustive mission list is provided by potoc. this mission list is the only source of truth about its mission.

similar to the core fellowship, one of potoc's goals is to recognize and retain developers who make substantial contributions to these areas. specifically, the collective should focus on people who have contributed to the specification or implementation of polkadot tooling. having an on-chain body will provide a means for these developers to align and organize.

potoc may want to apply for treasury funding in the future; it is therefore clear that any infringement upon its mission statement must be treated as an attempt to exfiltrate governance funds.

mission list

the mission list contains all projects and services that are maintained by potoc.

it is divided into two sections, solely to keep the requirements of each section clearly defined; core and dapp. it can be updated through an on-chain 2/3-majority rank-weighted vote across all members.

some obvious cornerstones that must be upheld by all projects are:

• must be foss (free and open-source software).
• source code must be accessible without a login requirement.
• all members of potoc must have permission to propose changes and raise issues.

the mission list will be defined as part of the initial seeding.

core tooling

generally, all tools that are used by the fellowship to craft, test, and validate updates fall into this category. but only the tools that are predominantly developed for this cause shall be maintained by potoc. for example, the rust language is not predominantly developed for polkadot; hence, it does not fit here.

dapp tooling

this category contains all tools and libraries that are used to build on polkadot and are predominantly developed for this cause.

structure

the initial structure shall be as minimalistic as possible, to limit maintenance and social drama. this has the downside of being less resilient against foreign takeovers, but in the beginning, i think it is fine to rather focus on social cohesion.

there is no differentiation between maintainers of "core tooling" and "dapp tooling". they both share the same set of members and rank structure. these terms are only used to clarify the mission goal.

rank 0 - candidate

developers who want to join can be inducted by any member into this rank. a vote for promotion to rank 1 can be started not earlier than six months after being inducted.

they are expected to be a maintainer for one of the tools on the mission list.

• voting weight: 0
• salary mult: task-based bounties

rank 1 - member

members are expected to keep up the maintenance of at least one tool from the mission list. this includes extending them and creating new tools that are needed by the polkadot developer community.

• voting weight: 1
• salary mult: 0.1 + task-based bounties

further ranks

the initial member seeding will show how many ranks need to be created to arrive at the correct granularity. the requirement for adding new ranks is that they are absolutely necessary.

we do not want to end in a situation where ranks are purely added to show off.

salary structure

there is no initial salary or bonus structure. in the future, after potoc proves itself worthy, it is proposed to set up a bounty-centered pay structure.

different tiers of bounties shall be defined, and a fairly low baseline salary will be paid to all members. this is done to provide an incentive for improving the tooling instead of sloth.

a member may apply for a specific bounty while providing evidence of fulfilling the bounty requirements. granting of a bounty will be decided upon by a 50% on-chain vote. a cap on the monthly bounty payout should be established to prevent runaway spending and prioritize important changes. bounty votes may only commence once per month to ease coordination between the members.

the exact constants for the size of the bounties and baseline salary will be proposed through a secondary document in phase ii.

initial members

there shall be a seeding repo that allows anyone to apply to a rank. applications should include a list of tools and contributions that show their past commitment to polkadot tooling development. the applicant must justify why they are a good fit to join potoc.

eventually, there needs to be a social consensus on the initial set of members. this set would then be proposed to the tokenholders, either as part of the potoc proposal itself or as a separate referendum.

future

there are a lot of questions unanswered by this document. in the beginning, we will use the core fellowship as a guiding light for processes and ethics as much as possible. eventually, potoc would develop its sub-culture (pun intended) and show more of its own identity.

a rough mid-term plan could look like this:

phase i - setup

this phase is for setting everything up and getting it enacted on-cain. it should take at most 6 months.

• 1.1 contacting all stakeholders: tokenholders, seeding members, and the fellowship.
• 1.2 pitching the proposal to the community and incorporating feedback.
• 2.1 crafting a preliminary implementation to ensure that all important parameters are mentioned in the proposal.
• 2.2 binding governance vote via wish-for-change track. this is treated as the final signal on whether to pursue it or not.
• 3.1 final implementation and approval by the fellowship.
• 3.2 enactment on-chain and beginning of operations.

phase ii - probation period

potoc starts operating in a risk-reduced mode without governance funding. this phase will be used to evaluate the effectiveness and impact of potoc.

hopefully, the general public will deem it useful and approve of its future funding requests.

this phase should take from 6 months to 1 year.

phase iii - business as usual

potoc enacts its salary structure. the creation and granting of bounties starts and attracts new developer talent.

this will demonstrate that creating a system collective can be done by anyone, and hopefully inspire the wider community to further its decentralization efforts in that direction.

a small and minimalistic website to monitor the compliance of the polkadot fellows available at fellowship.tasty.limo the code is on github.

architecture

it is written in and deployed to one of my personal servers. every 12 hours the data is fetched from an rpc provider. this is not ideal and should rather use light-clients, but i did not have the time to set that up; 80-20 you know.

the website is compacted into a single page to load within one http get call. this makes it very fast - even with jquery bloat.

the data extracted happens in three steps:

• loading all members from the polkadot-collectives parachain
• querying the on-chain identities from the polkadot relay
• fetching the github bio and checking if their address is mentioned

this is then rendered into a html template and a cache file for hot restarts.

future

i would like to decomission this eventually once the official website. but currently it is not showing the identities or github handles.

subweight is a cli to compare auto-generated rust config files of substrate. there can be thousands of those and manually reviewing them is too tedious. the tool available on crates.io and should work with all substrate blockchains.

prelude

substrate is a framework to write blockchains. it uses "weight" to annotate the estimated worst case resource consumption of an operation. this is crucial since it needs to ensure that all effort is paid for - we cannot "undo" the spent computational effort.

now these weights need to be updated every now and then. this happens automatically but should still be sanity checked by humans. this is where subweight comes into play. it breaks down ugly diffs and makes them graspable.

example

the files that it compares contain of lots of functions and look like this:

/// Storage: `Multisig::Multisigs` (r:1 w:1)
/// Proof: `Multisig::Multisigs` (`max_values`: None, `max_size`: Some(3346), added: 5821, mode: `MaxEncodedLen`)
/// Storage: `System::Account` (r:1 w:1)
/// Proof: `System::Account` (`max_values`: None, `max_size`: Some(128), added: 2603, mode: `MaxEncodedLen`)
/// The range of component `s` is `[2, 100]`.
/// The range of component `z` is `[0, 10000]`.
fn as_multi_complete(s: u32, z: u32, ) -> Weight {
	// Proof Size summary in bytes:
	//  Measured:  `392 + s * (33 ±0)`
	//  Estimated: `6811`
	// Minimum execution time: 45_986_000 picoseconds.
	Weight::from_parts(34_083_317, 0)
		.saturating_add(Weight::from_parts(0, 6811))
		.saturating_add(Weight::from_parts(159_314, 0).saturating_mul(s.into()))
		.saturating_add(Weight::from_parts(1_495, 0).saturating_mul(z.into()))
		.saturating_add(T::DbWeight::get().reads(2))
		.saturating_add(T::DbWeight::get().writes(2))
}

now, having about 1600 of those updated in a single merge request is impossibly to humanly review. subweight can be used in a few different ways to automatically show a comprehensible diff.

for example in merge request 129 it was used to plot a nice overview:

there is also a web version available, served by the binary subweight-web.

future

i hope to eventually retire this tool once we have either:

• a proper intermediate format to represent the formulas
• automatic metering that can abort execution and still charge fees

the second option is still a bit out, so for now it stays.

this website uses the zola static page generator and is themed with terminimal.

goals

all the things that my old website were not:

• simple to maintain
• aesthetic
• clear design
• (mostly) bloat free
• fast to load

architecture

in zola there are only two types of pages: indices and sections. one example for each would be the list of projects and this post. this means that all content needs to be split up into either of these categories. its quote simple once you understood to mange it.

a simple cli to lint and fix recurrent issues in rust workspaces surrounding 'features'. it can be installed with cargo install zepter and is documented here. it is most prominently used in the polkadot-sdk but can be used in any rust repo.

motivation

missing feature configuration can result in unexpected build errors that are very difficult to debug. especially when the dependency tree is deep.
we also kept getting external bug reports that our code would not build in downstream repos. resolving of these issues would take me about an hour; surely this was an unacceptable state.

now there are sadly many ways to mis-configure features in rust. zepter prevents about the three most common ones. it turns out that this was enough to cut the bug report rate down to zero. internal developers thanked me as well that this was a much welcomed change to reduce development time.

architecture

zepter extracts all relevant data from a rust workspace by utilizing cargo metadata. as user input it accepts a set of rules that must not be violated. it then creates a graph (directed-acyclic-graph) and tries to find violations of each rule. if that fails, the program succeeds.

usage

running the cli is enough to detect and fix mostly all issues in our code base. this is possible since polkadot-sdk has a config file that defines the default behaviour.

$ zepter 
[INFO] Running workflow 'default'
[INFO] 1/2 lint propagate-feature
[INFO] 2/2 format features

helpful error messages are provided in the ci through custom config:

Polkadot-SDK uses the Zepter CLI to detect abnormalities in the feature configuration.
It looks like one more more checks failed; please check the console output. You can try to automatically address them by running `zepter`.
Otherwise please ask directly in the Merge Request, GitHub Discussions or on Matrix Chat, thank you.

performance

the dag code is reasonably fast that a run in our codebase (488 crates) with multiple lints takes about 2 seconds. in general there can still be a lot of optimization, especially caching and not re-computing things multiple times. the code was just written very quickly (=optimized for dev speed :P).

the slowest part is the exhaustive path search in the dag, luckily this is embarrassingly parallel and could be optimized. the most memory intense operations are the generation of transitive closures of the dag. in most cases it is enough to build the transitive closure of a subset, which reduces memory consumption.