feat: use siphash for littDB sharding #1506
Conversation
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
There was a problem hiding this comment.
do we need this file ? or would it be generated during testing.
There was a problem hiding this comment.
All data beneath testdata is automatically generated. The purpose of all of these files is to simulate data migration from one format to another. I generated these files using the old serialization format in master, and use the code in this branch to read the data back again. The contents of this directory should be snapshots of the LittDB's file system taken at each version.
As to specifically whether we need this file, I'm not sure. It's generated internally by levelDB. I'm hoping to avoid having to hand prune these files.
litt/util/files.go
Outdated
| // The function performs a recursive copy of all files and directories, maintaining the same | ||
| // relative path structure and file metadata. If the destination directory exists, it will | ||
| // merge the source content into it, potentially overwriting files with the same names. | ||
| func CopyDirectoryRecursively(src, dst string) error { |
There was a problem hiding this comment.
this assumes the dst is writable right ? Should we check if it is and error out ?
There was a problem hiding this comment.
I added this check. I also added some unit tests to verify.
litt/util/hashing.go
Outdated
| } | ||
| hash, err := hasher.Write(key) | ||
| if err != nil { | ||
| panic(err) |
There was a problem hiding this comment.
Nope! Place holder from when I initially wrote this, but I forgot to circle back and fix it. This method now returns an error.
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
litt/util/hashing.go
Outdated
| "encoding/binary" | ||
| "fmt" | ||
|
|
||
| "github.com/aead/siphash" |
There was a problem hiding this comment.
im wondering if we should use this lib of siphash https://github.com/dchest/siphash
Signed-off-by: Cody Littley <[email protected]>
litt/disktable/segment/segment.go
Outdated
|
|
||
| hash, err := util.HashKey(key, s.metadata.salt) | ||
| if err != nil { | ||
| return 0, fmt.Errorf("failed to hash key: %v", err) |
There was a problem hiding this comment.
%v -> %w (also elsewhere in this PR)
There was a problem hiding this comment.
I replaced a bunch of %vs in segment.go (many of which were not introduced in this PR), but didn't find any others.
litt/disktable/disk_table.go
Outdated
| metadata.GetShardingFactor(), | ||
| config.SaltShaker.Uint32(), | ||
| ([16]byte)(salt), |
litt/util/files.go
Outdated
| // The function checks that the destination has appropriate write permissions before starting the copy. | ||
| // If the destination directory doesn't exist, it verifies the parent directory has appropriate permissions. | ||
| // For existing directories, it ensures they have write permissions before attempting to copy files into them. | ||
| func CopyDirectoryRecursively(source string, destination string) error { |
There was a problem hiding this comment.
nit: would be nice to have all exported functions at the top, so it's clear what is "important" here
Signed-off-by: Cody Littley <[email protected]>
Signed-off-by: Cody Littley <[email protected]>
* use siphash for sharding Signed-off-by: Cody Littley <[email protected]> * Start work on migration test. Signed-off-by: Cody Littley <[email protected]> * Generate table using new format Signed-off-by: Cody Littley <[email protected]> * Added migration test, need to debug Signed-off-by: Cody Littley <[email protected]> * fix migration Signed-off-by: Cody Littley <[email protected]> * made suggested changes Signed-off-by: Cody Littley <[email protected]> * Made suggested changes. Signed-off-by: Cody Littley <[email protected]> * Change siphash implementation. Signed-off-by: Cody Littley <[email protected]> * made suggested changes Signed-off-by: Cody Littley <[email protected]> * fix go.sum Signed-off-by: Cody Littley <[email protected]> --------- Signed-off-by: Cody Littley <[email protected]>
Why are these changes needed?
Switch to siphash for littDB sharding.