chore(dot/types): integrate new scale pkg into dot/types (ChainSafe#1817)

* WIP/Use vdt slice in header

* WIP/header changes

* scale fix

* Use vdts in header

* TODO/Try and migrate away from using common.optional

* test optional header encoding

* scale encode and decode for block data type

* encode and decode block data slices

* WIP/fix slice decoding

* pass tests for block data

* rewrite block data tests

* remove old funcs in blockdata

* wip

* fix scale tests

* TODO/integrate header changes

* mew block data in dot state

* use new header in block type

* add grandpa type with new header

* remove stuff from optional types

* clean up

* small cleanup

* WIP/refactor consensus digest

* VDT consensus digest works, have to fix digst tests to not use types

* add consensus digest testing

* refactor to use new NextEpochData type

* refactor babe disabled

* refactor babe consensus digest types to use Vdts

* grandpa refactor forced change to use Vdts

* refactor grandpa pause and resume to use Vdts

* refactor digest to use new Vdts for consensus

* change tests

* fix tests

* updated tests

* clean up and fix tests

* cleanup

* fix tests

* begin removing duplicate types and methods used for testing

* remove consensus digest byte types

* fix runtime tests

* cleanup digest

* clean up digest

* remove test methods that arent needed

* move consensus digest vdts to constructor

* move new digest types to constructor

* block announce refactor

* WIP/network messages refactor

* WIP/message refactor

* new types and tests for network messages

* add interface methods

* fix network message tests

* minor fix to network message tests

* integrate a large amount of new message type and pass tests

* wip

* integrate scale into network transaction

* WIP/light message refactor

* remove light message changes

* big wip

* wip

* wip, tests failing

* digest tests pass

* remove comments from digest test

* WIP

* wip/fix new build block method

* wip

* wip/debug

* wip/fix sync tests

* fix block encoding issue in runtime ExecuteBlock

* fix dot core tests

* fix dot state tests

* fix grandpa tests

* fix babe tests

* fix rpc tests

* pass stress tests locally

* fix an rpc test

* more test fixes

* update tests

* clean up dot digest

* start removing old blockstate methods

* pass stress tests

* fix block race test

* more test fixes

* remove old babe methods

* replace old block instances in runtime

* fixes

* remove old execute block instances

* wip

* remove execute block from runtime

* remove old block type

* clean up and fix naming for block type

* remove old finalization info and fix naming

* wip/header integration

* clean up for PR

* wip

* wip/remove old header instances

* temp fix runtime interface

* fix runtime inteface

* fix runtime errors

* fix stress tests

* fix babe and grandpa

* use old names for epoch state functions

* fix names for blockstate functions

* fix changes for tests

* wip/get tests passing

* fix naming and mocks for runtime functions

* continue cleaning up code and removing old methods

* change header naming and continue to rename old methods

* fix sync mock

* fix mock types

* continue fix tests

* remove old digest logic

* wip/rename digest functions

* renaming digest

* rename and clean block data

* fix scale

* clean up

* clean up code for review

* fix and clean

* wip/lint

* linted

* try and fix test

* fix tests and linting

* fix mock formatting

* refactor types package

* refactor types package

* chore (pkg/scale) types refactor to utilize new scale pkg

* digest refactor tests pass

* refactor dot/digest to better use vdts

* remove consensus data type function

* remove unnecessary logs in digest test

* remove logs from digest test

* remove log from block announce

* Update dot/types/babe.go

Co-authored-by: noot <[email protected]>

* add comment to block data

* Update dot/types/block_data_test.go

Co-authored-by: noot <[email protected]>

* Update dot/import.go

Co-authored-by: Timothy Wu <[email protected]>

* WIP/cr feedback

* use typecast in getEpochForBlock

* use typecast in babe

* remove new empty Block function

* typecast and error handling in babe/verify

* condense new blockData func

* handle errors for vdt.Add

* error out on error in database test

* return error in handleBlock

* and function to test if header is empty

* lint

* remove switch from handleDigests

* tests for new block request

* remove optionals from the network

* add err to return type for deep copy

* cr feedback

* lint

* fix tests

* remove log from digest

* wip/fix tests

* refactor types package to use new scale

* wip/integrate scale into grandpa

* remove old funcs

* grandpa message type refactor

* remove old scale from grandpa message

* clean up and remove scale from message handler

* remove scale from message handler test

* remove scale from grandpa

* remove type methods

* remove old scale from grandpa

* wip/debug grandpa

* wip

* wip/partial integraitons and tests pass

* wip/tests passing

* wip/tests passing

* wip/remove grandpaVoter from state and tests pass

* fixed grandpa bug finally

* fix naming in grandpa types

* fix remaining conflicts

* remove final conflict

* lint and clean up

* fix linting

* lint

* cr feedback

* make newGrandpaMessage private

* cr feedback

* remove scale from body

* remove asOptional method from body

* remove scale from body tests

* remove scale from inherents

* fix TestGrandpaAuthoritiesRawToAuthorities test

* remove decode for raw auths

* remove scale from types

* make babe digest a VDT

* remove unneeded functions from babe digest

* fix babe error

* remove unneeded functions from dot/types

* lint

* fix babe verification

* cr feedback

* Update dot/types/inherents.go

Co-authored-by: noot <[email protected]>

* rename scale voter

* renaming

* remove getters and interface from babe digest

* remove exp encoding from block test because it wrong

* remove repeated logic

* lint

* lint

* change naming

* cr feedback

* cr feedback

* chore(dot/state): integrate scaleV2 into state (ChainSafe#1822)

* chore(lib/common): remove the common/optional pkg (ChainSafe#1879)

* remove old scale from dot state

* add error handling to ToPreRuntimeDigest

* fix diff

* fix errors

* lint

* revise APIItems

* update scale integration for version

* wip/remove optionalBytes from runtime

* remove optinal uint32

* merge with scale runtime integration branch

* remove optional bytes from life/resolver and its tests

* remove optional Bytes from runtime/life

* remove encoding and decoding of optional bytes

* remove not used optional functions

* refactor light message

* refactor lightRequest

* remove unused types from common/optional

* lint

* remove optionals

* remove final optional types and lint

* CR feedback

* lint

* remove todo

* remove light request attribute names

* remove light response attribute names

* lint

Co-authored-by: Tim Wu <[email protected]>

* wip/CR feedback

* lint and maybe bug fix

* lint

* fix fmt

* wip/fix wasmer tests

* remove comments

* fix wasmer tests

Co-authored-by: Tim Wu <[email protected]>
Co-authored-by: noot <[email protected]>
Co-authored-by: noot <[email protected]>

Author: 3 people

dot/core/messages_test.go

@@ -92,7 +92,9 @@ func TestService_HandleBlockProduced(t *testing.T) {
 
 	// simulate block sent from BABE session
 	digest := types.NewDigest()
-	err = digest.Add(*types.NewBabeSecondaryPlainPreDigest(0, 1).ToPreRuntimeDigest())
+	prd, err := types.NewBabeSecondaryPlainPreDigest(0, 1).ToPreRuntimeDigest()
+	require.NoError(t, err)
+	err = digest.Add(*prd)
 	require.NoError(t, err)
 
 	newBlock := types.Block{

dot/core/service_test.go

@@ -82,7 +82,9 @@ func TestAnnounceBlock(t *testing.T) {
 
 	// simulate block sent from BABE session
 	digest := types.NewDigest()
-	err = digest.Add(*types.NewBabeSecondaryPlainPreDigest(0, 1).ToPreRuntimeDigest())
+	prd, err := types.NewBabeSecondaryPlainPreDigest(0, 1).ToPreRuntimeDigest()
+	require.NoError(t, err)
+	err = digest.Add(*prd)
 	require.NoError(t, err)
 
 	newBlock := types.Block{

dot/digest/digest_test.go

@@ -367,14 +367,19 @@ func TestHandler_HandleBABEOnDisabled(t *testing.T) {
 }
 
 func createHeaderWithPreDigest(t *testing.T, slotNumber uint64) *types.Header {
-	babeHeader := types.NewBabePrimaryPreDigest(0, slotNumber, [32]byte{}, [64]byte{})
+	t.Helper()
 
-	enc := babeHeader.Encode()
+	babeHeader := types.NewBabeDigest()
+	err := babeHeader.Set(*types.NewBabePrimaryPreDigest(0, slotNumber, [32]byte{}, [64]byte{}))
+	require.NoError(t, err)
+
+	enc, err := scale.Marshal(babeHeader)
+	require.NoError(t, err)
 	d := &types.PreRuntimeDigest{
 		Data: enc,
 	}
 	digest := types.NewDigest()
-	err := digest.Add(*d)
+	err = digest.Add(*d)
 	require.NoError(t, err)
 
 	return &types.Header{