Adds multi-block election types and refactors current pallets to support new interfaces and types #6034

gpestana · 2024-10-13T23:49:28Z

This PR refactors the types and structs required to run a mulit-block election and updates the EPM, staking-pallet and all dependent pallets to use the multi-block types. The Westend runtime is configured to run a 1 paged election, which is a noop refactor compared to the current single-block election.

Note: The multi-block election provider pallet is wip and it's added in a separate PR (#6213).

To finish:

add benchmarks for on_intialize in pallet staking

substrate/frame/election-provider-support/src/lib.rs

Ank4n

Only looked at traits. Will do another pass.

substrate/frame/election-provider-support/src/lib.rs

Ank4n · 2024-10-15T13:49:23Z

substrate/frame/election-provider-support/src/lib.rs

+	fn elect(remaining: PageIndex) -> Result<BoundedSupportsOf<Self>, Self::Error>;
+
+	/// The index of the *most* significant page that this election provider supports.
+	fn msp() -> PageIndex {


may be call it highest page or top page?

Also, since the current page index needs to be stored somewhere, could we just store it on ElectionProvider, and Staking then can just call next_elect()?

I think I called it like this initially be be simiar to the concept if msp (most significant bit) in cpus.

…into gpestana/epm-mb

substrate/frame/election-provider-support/src/onchain.rs

substrate/frame/election-provider-support/src/lib.rs

kianenigma · 2024-10-24T16:38:53Z

substrate/frame/election-provider-support/src/onchain.rs

+		// defensive: Since npos solver returns a result always bounded by `desired_targets`, this
+		// is never expected to happen as long as npos solver does what is expected for it to do.
+		let supports: BoundedSupportsOf<Self> = to_supports(&staked)
+			.try_into_bounded_supports()


okay, so this conversion checks the bounds, and bails if any are not met. So the onchain election has no trimming capabiliites here.

Here this is fine. But long term the MB-Miner should ideally be able to trim to meet these bounds, please make a note of this if it is not the case already.

Keeping track of this here #6237, I'll ensure that's the case for all the (offchain) miners.

…ave exposure gaps in pages)

Ank4n · 2024-10-28T13:56:14Z

substrate/frame/election-provider-multi-phase/src/lib.rs

@@ -652,20 +669,22 @@ pub mod pallet {
 		#[pallet::constant]
 		type SignedDepositWeight: Get<BalanceOf<Self>>;

-		/// The maximum number of winners that can be elected by this `ElectionProvider`
-		/// implementation.
+		/// Maximum number of winners that an electio supports.


Suggested change

/// Maximum number of winners that an electio supports.

/// Maximum number of winners that an election supports.

Ank4n · 2024-10-28T13:58:41Z

substrate/frame/election-provider-multi-phase/src/lib.rs

-	}
+	fn elect(page: PageIndex) -> Result<BoundedSupportsOf<Self>, Self::Error> {
+		// Note: this pallet **MUST** only by used in the single-block mode.
+		ensure!(page.is_zero(), ElectionError::<T>::MultiPageNotSupported);


BadPageIndex or similar makes more sense I think

Suggested change

ensure!(page.is_zero(), ElectionError::<T>::MultiPageNotSupported);

ensure!(page == SINGLE_PAGE, ElectionError::<T>::MultiPageNotSupported);

kianenigma · 2024-11-05T12:48:21Z

substrate/frame/election-provider-multi-phase/src/lib.rs

@@ -1398,7 +1413,8 @@ impl<T: Config> Pallet<T> {
 	/// Current best solution, signed or unsigned, queued to be returned upon `elect`.
 	///
 	/// Always sorted by score.
-	pub fn queued_solution() -> Option<ReadySolution<T::AccountId, T::MaxWinners>> {
+	pub fn queued_solution(
+	) -> Option<ReadySolution<T::AccountId, T::MaxWinners, T::MaxBackersPerWinner>> {


Might be time to make this ReadySolution<T> to avoid mixing the latter two.

kianenigma · 2024-11-05T12:50:07Z

substrate/frame/election-provider-multi-phase/src/lib.rs

@@ -1663,7 +1681,9 @@ impl<T: Config> Pallet<T> {
 	}

 	/// record the weight of the given `supports`.
-	fn weigh_supports(supports: &Supports<T::AccountId>) {
+	fn weigh_supports(
+		supports: &BoundedSupports<T::AccountId, T::MaxWinners, T::MaxBackersPerWinner>,


Same here.

Not urgently needed. But the risk is if we mix the order..

kianenigma · 2024-11-05T12:53:47Z

substrate/frame/election-provider-multi-phase/src/lib.rs

+			assert!(QueuedSolution::<Runtime>::get().is_some());
+
+			// single page elect call works as expected.
+			assert_ok!(MultiPhase::elect(SINGLE_PAGE));


If you call MultiPhase::elect(SINGLE_PAGE) twice in a row, it will fail anyways. So the second call is not being treated fairly, you should first assert that MultiPhase::elect(10) is a noop and returns an error, then try MultiPhase::elect(SINGLE_PAGE). Or, do the MultiPhase::elect(SINGLE_PAGE) first, but in a transactional layer and rollback.

kianenigma · 2024-11-05T12:54:18Z

substrate/frame/election-provider-multi-phase/src/lib.rs

-					(40, Support { total: 60, voters: vec![(2, 5), (3, 10), (4, 5), (40, 40)] })
-				]
-			);
+			let expected_supports = vec![


comparing bounded stuff is a PITA..

Ank4n

This is a long PR. Haven't gone through everything yet.

Ank4n · 2024-11-05T10:51:13Z

prdoc/pr_6034.prdoc

+doc:
+  - audience: Runtime Dev
+    description: |
+      This PR adds election types and structs required to run a multi-block election. In additoin,


Suggested change

This PR adds election types and structs required to run a multi-block election. In additoin,

This PR adds election types and structs required to run a multi-block election. In addition,

Ank4n · 2024-11-05T10:55:49Z

substrate/frame/election-provider-multi-phase/src/lib.rs

+//! [`frame_election_provider_support::ElectionProvider`] traits used by this pallet support a
+//! multi page election.


Suggested change

//! [`frame_election_provider_support::ElectionProvider`] traits used by this pallet support a

//! multi page election.

//! [`frame_election_provider_support::ElectionProvider`] traits used by this pallet can support a

//! multi-page election.

Ank4n · 2024-11-05T10:58:42Z

substrate/frame/election-provider-multi-phase/src/lib.rs

+//! [`frame_election_provider_support::ElectionProvider`] traits used by this pallet support a
+//! multi page election.
+//!
+//! However, this pallet is meant to be used only in the context of a single-page election and data


Suggested change

//! However, this pallet is meant to be used only in the context of a single-page election and data

//! However, this pallet only supports single-page election and data

Ank4n · 2024-11-05T11:11:20Z

substrate/frame/election-provider-multi-phase/src/lib.rs

@@ -494,13 +508,15 @@ pub enum ElectionError<T: Config> {
 	DataProvider(&'static str),
 	/// An error nested in the fallback.
 	Fallback(FallbackErrorOf<T>),
+	/// An error occurred when requesting an election result. The caller expects a mulit-paged
+	/// election, which this pallet does not support.
+	MultiPageNotSupported,


Prefer UnsupportedPageIndex which is more general. When this/another pallet supports pages, it could still run into conceptually similar error.

Ank4n · 2024-11-05T11:12:09Z

substrate/frame/election-provider-multi-phase/src/lib.rs

 		type MaxWinners: Get<u32>;

+		/// Maximum number of voters that can support a winner target in an election solution.


Suggested change

/// Maximum number of voters that can support a winner target in an election solution.

/// Maximum number of voters that can support a winner in an election solution.

Ank4n · 2024-11-05T11:41:21Z

substrate/frame/election-provider-multi-phase/src/unsigned.rs

@@ -427,8 +429,10 @@ pub trait MinerConfig {
 	///
 	/// The weight is computed using `solution_weight`.
 	type MaxWeight: Get<Weight>;
-	/// The maximum number of winners that can be elected.
+	/// The maximum number of winners that can be elected per page (and  overall).


overall? Since this pallet only supports one page? Bit confusing.

Ank4n · 2024-11-05T11:42:21Z

substrate/frame/election-provider-multi-phase/src/unsigned.rs

 	type MaxWinners: Get<u32>;
+	/// The maximum number of backers (edges) per winner in the last solution.


Just to try using less new terminologies

Suggested change

/// The maximum number of backers (edges) per winner in the last solution.

/// The maximum number of backers per winner in the last solution.

Ank4n · 2024-11-05T11:47:40Z

substrate/frame/election-provider-support/src/lib.rs

+//! Both [`ElectionDataProvider`] and [`ElectionProvider`] traits are parameterized by page,
+//! supporting an election to be performed over multiple pages. This enables the
+//! [`ElectionDataProvider`] implementor to provide all the election data over multiple pages.
+//! Similarly [`ElectionProvider::elect`] is parameterized by page index.


Does election data and elect pages indexed same? Or can one have more pages than the other?

Ank4n · 2024-11-05T11:53:11Z

substrate/frame/election-provider-support/src/lib.rs

@@ -293,24 +326,32 @@ pub trait ElectionDataProvider {
 	/// Maximum number of votes per voter that this data provider is providing.
 	type MaxVotesPerVoter: Get<u32>;

-	/// All possible targets for the election, i.e. the targets that could become elected, thus
-	/// "electable".
+	/// Returns the possible targets for the election associated with page `page`, i.e. the targets


Suggested change

/// Returns the possible targets for the election associated with page `page`, i.e. the targets

/// Returns the possible targets for the election associated with the provided `page`, i.e. the targets

Should we also add here the expectation that it should return targets sorted such that the highest page (msp) returns the top targets?

Ank4n · 2024-11-05T12:53:04Z

substrate/frame/election-provider-support/src/lib.rs

+
+/// An extension trait to convert from [`sp_npos_elections::Supports<AccountId>`] into
+/// [`BoundedSupports`].
+pub trait TryIntoBoundedSupports<AccountId, BOuter: Get<u32>, BInner: Get<u32>> {


I think probably because we need to somehow pass the bounds. But I still feel the trait is not very useful, since you should be able to build BoundedSupports only from Supports.

I would remove the trait, and have a fn implemented directly on the type Supports.

kianenigma · 2024-11-05T13:22:20Z

substrate/frame/staking/src/lib.rs

+#[derive(PartialEq, Eq, Clone, Encode, Decode, RuntimeDebug, TypeInfo, MaxEncodedLen)]
+pub enum SnapshotStatus<AccountId> {
+	/// Paged snapshot is in progress, the `AccountId` was the last staker iterated.
+	Ongoing(AccountId),


You can use the #[default] attribite here.

kianenigma · 2024-11-05T13:25:50Z

substrate/frame/staking/src/lib.rs

@@ -382,6 +414,13 @@ pub struct ActiveEraInfo {
 	pub start: Option<u64>,
 }

+/// Pointer to the last iterated indices for targets and voters used when generating the snapshot.
+#[derive(Encode, Decode, RuntimeDebug, TypeInfo, MaxEncodedLen)]
+pub(crate) struct LastIteratedStakers<AccountId> {


kianenigma · 2024-11-05T13:30:06Z

substrate/frame/staking/src/pallet/mod.rs

+				<<T as Config>::ElectionProvider as ElectionProvider>::Pages::get().into();
+
+			// election ongoing, fetch the next page.
+			if let Some(started_at) = ElectingStartedAt::<T>::get() {


This storage item is never set?

kianenigma · 2024-11-05T13:37:07Z

substrate/frame/staking/src/pallet/impls.rs

+	///
+	/// Deduplicates stashes in place and returns an error if the bounds are exceeded.
+	pub(crate) fn add_electables(
+		stashes: BoundedVec<T::AccountId, MaxWinnersPerPageOf<T::ElectionProvider>>,


check in integrity_test: MaxWinnersPerPageOf * PageCount <= ValidatorSetSize

kianenigma · 2024-11-05T13:39:41Z

substrate/frame/staking/src/pallet/mod.rs

+	/// While the lock is set, there should be no mutations on the ledgers/staking data, ensuring
+	/// that the data provided to [`Config::ElectionDataProvider`] is stable during all pages.
+	#[pallet::storage]
+	pub(crate) type ElectionDataLock<T: Config> = StorageValue<_, (), OptionQuery>;


Do we need this? if someone changes their stake mid snapshot, what could go wrong?

Okay the only thing is that we should be worried about is removing nodes form teh VoteList during this, so that the iteration can resume at a reasonable spot.

To be specific, we can remove anyone, other than the single poor voter/target that is the pointer to the next page.

I suggest we add this logic directly to VoterList via some hook, and any operation like chill that attempts to remove a node might fail on T::VoterList::remove()?, so no extra code is needed here to handle the error.

We certainly won't need a storage item here. We just need to communicate to the VoterList like:

/// Voter list pallet needs a single storage item, that is one `node` that cannot be removed. #[pallet::storage] pub type IterationLock = StoarageValue<_> /// set/uptess `who` as the lock prevents them from being removed from the list /// This should be called per non-terminal page of snapshot creation T::VoterList::set_lock(who) /// Remove any existing locks /// This is called on the terminal (last) page of the snapshot creation. T::VoterList::remove_lock()

kianenigma · 2024-11-05T13:47:37Z

substrate/frame/staking/src/pallet/impls.rs

+	pub(crate) fn add_electables(
+		stashes: BoundedVec<T::AccountId, MaxWinnersPerPageOf<T::ElectionProvider>>,
+	) -> Result<(), ()> {
+		let mut storage_stashes = ElectableStashes::<T>::get();


This can be written better with a ElectableStashes::mutate.

kianenigma · 2024-11-05T13:49:20Z

substrate/primitives/staking/src/lib.rs

@@ -461,6 +499,42 @@ pub struct PagedExposureMetadata<Balance: HasCompact + codec::MaxEncodedLen> {
 	pub nominator_count: u32,
 	/// Number of pages of nominators.
 	pub page_count: Page,
+	/// Number of empty slots in the last page.
+	pub last_page_empty_slots: u32,


@Ank4n please review the chnages related to this.

gpestana added 4 commits October 13, 2024 15:34

init: adds EPM-MB, multipaged types and primitives

3f6bf3f

refactors staking pallet with new paginated types

8a682b8

refactors election-multiphase pallet with new paginated types

94ff073

nits for dependent pallets and runtimes

1527bd6

gpestana self-assigned this Oct 13, 2024

gpestana requested a review from a team as a code owner October 13, 2024 23:49

gpestana requested a review from Ank4n October 13, 2024 23:49

gpestana marked this pull request as draft October 13, 2024 23:49

gpestana added the T2-pallets This PR/Issue is related to a particular pallet. label Oct 13, 2024

gpestana mentioned this pull request Oct 14, 2024

WIP: monitor command for the multi-block staking miner paritytech/polkadot-staking-miner#910

Draft

4 tasks

adds ElectionProvider::ongoing to trait and impls

5cb1de1

gpestana mentioned this pull request Oct 14, 2024

Introduces multi-block election types and traits and refactors relevant pallets #5438

Closed

Merge branch 'master' into gpestana/epm-mb

50d9698

Ank4n reviewed Oct 15, 2024

View reviewed changes

substrate/frame/election-provider-support/src/lib.rs Show resolved Hide resolved

Ank4n reviewed Oct 15, 2024

View reviewed changes

gpestana added 8 commits October 18, 2024 12:44

clean up sp-staking

43eec4b

clean up and rustdoc for epm-support

11dd785

Merge branch 'gpestana/epm-mb' of github.com:paritytech/polkadot-sdk …

d446ed0

…into gpestana/epm-mb

nits onchain election provider

5c17573

EPM nits and rustdoc improvements

ee7bdd4

clean up and rustdocs for EPM core and sub pallets

f30b548

rustdocs and nits

e7bf7dc

Merge branch 'master' into gpestana/epm-mb

f810692

gpestana marked this pull request as ready for review October 18, 2024 16:42

gpestana added 2 commits October 18, 2024 18:57

adds prdoc

bf67a3e

fixes prdoc

345840d

removes from all pallets

52dcee9

kianenigma reviewed Oct 24, 2024

View reviewed changes

substrate/frame/election-provider-support/src/onchain.rs Show resolved Hide resolved

kianenigma reviewed Oct 24, 2024

View reviewed changes

substrate/frame/election-provider-support/src/onchain.rs Show resolved Hide resolved

kianenigma reviewed Oct 24, 2024

View reviewed changes

substrate/frame/election-provider-support/src/lib.rs Show resolved Hide resolved

kianenigma reviewed Oct 24, 2024

View reviewed changes

gpestana added 2 commits October 25, 2024 14:17

Merge branch 'master' into gpestana/epm-mb

18813c9

Addresses PR reviews (removes all paged types/configs from epm and nits)

a92d44a

gpestana mentioned this pull request Oct 25, 2024

Election miner implementation should be able to trim an election result to fit into the expected bounds #6237

Open

improve memory efficiency of try into bounded supports; tests

3d1a330

gpestana requested a review from kianenigma October 25, 2024 15:17

gpestana added 2 commits October 31, 2024 15:54

a few nits in the staking pallet; handles exposures better (do not le…

1757c91

…ave exposure gaps in pages)

updates electable stashes to allow for multiple adds across pages

7022467

gpestana mentioned this pull request Nov 1, 2024

Design and implement the lockable election provider for multi-block election #6328

Open

Adds staking pallet rustdocs; nits

b49684e

Ank4n reviewed Nov 1, 2024

View reviewed changes

This was referenced Nov 4, 2024

Add size bounds to pallet election-provider-multi-phase #6348

Draft

[tracking] remove without_storage_info for pallets migrating to AH #6289

Open

kianenigma reviewed Nov 5, 2024

View reviewed changes

Ank4n reviewed Nov 5, 2024

View reviewed changes

gpestana requested a review from georgepisaltu November 5, 2024 13:19

paritytech-review-bot bot requested a review from a team November 5, 2024 13:20

kianenigma reviewed Nov 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds multi-block election types and refactors current pallets to support new interfaces and types #6034

Adds multi-block election types and refactors current pallets to support new interfaces and types #6034

gpestana commented Oct 13, 2024 •

edited

Loading

Ank4n left a comment

Ank4n Oct 15, 2024

Ank4n Oct 15, 2024

kianenigma Oct 24, 2024

kianenigma Oct 24, 2024

gpestana Oct 25, 2024

Ank4n Oct 28, 2024

Ank4n Oct 28, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

Ank4n left a comment

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

Ank4n Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024

kianenigma Nov 5, 2024 •

edited

Loading

kianenigma Nov 5, 2024

	/// Maximum number of winners that an electio supports.
	/// Maximum number of winners that an election supports.

	ensure!(page.is_zero(), ElectionError::<T>::MultiPageNotSupported);
	ensure!(page == SINGLE_PAGE, ElectionError::<T>::MultiPageNotSupported);

	This PR adds election types and structs required to run a multi-block election. In additoin,
	This PR adds election types and structs required to run a multi-block election. In addition,

		//! [`frame_election_provider_support::ElectionProvider`] traits used by this pallet support a
		//! multi page election.

	//! However, this pallet is meant to be used only in the context of a single-page election and data
	//! However, this pallet only supports single-page election and data

		type MaxWinners: Get<u32>;

		/// Maximum number of voters that can support a winner target in an election solution.

		type MaxWinners: Get<u32>;
		/// The maximum number of backers (edges) per winner in the last solution.

	/// The maximum number of backers (edges) per winner in the last solution.
	/// The maximum number of backers per winner in the last solution.

	/// Returns the possible targets for the election associated with page `page`, i.e. the targets
	/// Returns the possible targets for the election associated with the provided `page`, i.e. the targets

Adds multi-block election types and refactors current pallets to support new interfaces and types #6034

Are you sure you want to change the base?

Adds multi-block election types and refactors current pallets to support new interfaces and types #6034

Conversation

gpestana commented Oct 13, 2024 • edited Loading

Ank4n left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ank4n left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gpestana commented Oct 13, 2024 •

edited

Loading

kianenigma Nov 5, 2024 •

edited

Loading