Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions doc/manual/rl-next/nix-store-import-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
synopsis: Add `nix store import` and `nix store export`
prs: #9474
issues: #9038
description: {

Added a pair of new commands: [`nix store import`] and [`nix store export`].

[`nix store export`]: @docroot@/command-ref/new-cli/nix3-store-export.md
[`nix store import`]: @docroot@/command-ref/new-cli/nix3-store-import.md

}



15 changes: 15 additions & 0 deletions src/nix/store-export.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
R""(

# Examples

* Export the closure of a given installable and re-import it in another machine

```console
$ nix store export --recursive --format binary nixpkgs#hello > hello-closure.tar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suggests that the output is a tarball, which is not the case.

$ ssh user@otherHost nix store import < hello-closure.tar
```

# Description

This command generates an archive containing the serialisation of *installable*, as well as all the metadata required so that it can be imported with [`nix store import`](@docroot@/command-ref/new-cli/nix3-store-import.md).
)""
134 changes: 134 additions & 0 deletions src/nix/store-import.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
#include "command.hh"
#include "store-api.hh"

using namespace nix;

struct MixImportExport : virtual Args
{
enum struct ArchiveFormat
{
Binary,
};

std::optional<ArchiveFormat> format = std::nullopt;
MixImportExport()
{
addFlag(
{.longName = "format",
.description = R"(
Format of the archive.
The only supported format is `binary`, which corresponds to the format used by [`nix-store --export`](@docroot@/command-ref/nix-store/export.md).
)",
.labels = {"format"},
.handler = {[&](std::string_view arg) {
if (arg == "binary")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally the format identifier is usable out of context, because it's more recognizable and searchable that way.
E.g.

Suggested change
if (arg == "binary")
if (arg == "nix-store-object-stream-0")

thought: For version 1 it would be wise to add magic bytes. That would bring it up to NAR level, which has nix-archive-1 for magic.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Erk, I really wouldn't want to have to write nix store export --format nix-store-object-stream-0 😬 (at least not if there's no alternative)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe binary should be an alias for this format name, at which point we're almost full circle. I think that's good?

Copy link
Contributor

@fricklerhandwerk fricklerhandwerk Nov 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could lay groundwork to be able to say in the future something like --format binary --format-version 1 (default to latest). But I agree with @roberth that there should be a "fancy" or fully-qualified, distinctive name that is easy to find when one needs to figure things out.

Copy link
Member Author

@thufschmitt thufschmitt Nov 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we can add that, but isn't that vastly overkill and confusing for something that we most likely won't want to evolve? We might add new formats, which is why we settled on having this --format parameter, but if we ever add another one, that'll most likely be something totally different. And I really don't see anyone willing to pass --format nix-store-object-stream-0

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can imagine this, but actually we can add the behavior I proposed exactly when needed. Therefore nevermind.

format = ArchiveFormat::Binary;
else
throw Error("Unknown archive format: %s", arg);
}}});
}

};

struct CmdStoreExport : StorePathsCommand, MixImportExport
{
std::string description() override
{
return "Export the given store path(s) in a way that can be imported by `nix store import`.";
}

std::string doc() override
{
return
#include "store-export.md"
;
}

std::optional<Path> outputFile = std::nullopt;

CmdStoreExport()
{
addFlag({
.longName = "output-file",
.description = "Write the archive to the given file instead of stdout.",
.labels = {"file"},
.handler = {&outputFile},
});

}

void run(ref<Store> store, StorePaths && storePaths) override
{

// We don't use the format parameter now, but we still want to enforce it
// being specified to not block us on backwards-compatibility.
if (!format)
throw UsageError("`--format` must be specified");

StorePathSet pathsAsSet;
pathsAsSet.insert(storePaths.begin(), storePaths.end());

auto sink = [&]() -> FdSink {
if (outputFile) {
if (*outputFile == "-") {
return FdSink(STDOUT_FILENO);
} else {
auto fd = open(outputFile->c_str(), O_CREAT | O_WRONLY | O_TRUNC, 0644);
return FdSink(fd);
}
} else if (isatty(STDOUT_FILENO)) {
throw Error("Refusing to write binary data to a terminal. Use `--output-file` to specify a file to write to.");
} else {
return FdSink(STDOUT_FILENO);
}
}();

store->exportPaths(pathsAsSet, sink);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we've named the protocol, we can mention it in the exportPaths method's header, and link to the manual.

sink.flush();
}
};

struct CmdStoreImport : StoreCommand, MixImportExport
{
std::string description() override
{
return "Import the given store path(s) from a file created by `nix store export`.";
}
std::string doc() override
{
return
#include "store-import.md"
;
}

std::optional<Path> inputFile = std::nullopt;

CmdStoreImport() {
addFlag({
.longName = "input-file",
.description = "Read the archive from the given file instead of stdin.",
.labels = {"file"},
.handler = {&inputFile},
});
}

void run(ref<Store> store) override
{
FdSource source = [&]() -> FdSource {
if (inputFile && *inputFile != "-") {
auto fd = open(inputFile->c_str(), O_RDONLY);
return FdSource(fd);
} else {
return FdSource(STDIN_FILENO);
}
}();
auto paths = store->importPaths(source, NoCheckSigs);

for (auto & path : paths)
logger->cout("%s", store->printStorePath(path));
}

};

static auto rStoreExport = registerCommand2<CmdStoreExport>({"store", "export"});
static auto rStoreImport = registerCommand2<CmdStoreImport>({"store", "import"});
16 changes: 16 additions & 0 deletions src/nix/store-import.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
R""(

# Examples

* Import a closure that has been exported from another machine

```console
$ ssh user@otherHost nix store export --recursive --format binary nixpkgs#hello > hello-closure.tar
$ nix store import < hello-closure.tar
```

# Description

This command reads an archive of store paths, as produced by [`nix store export`](@docroot@/command-ref/new-cli/nix3-store-export.md), and adds it to the store.

)""
1 change: 1 addition & 0 deletions tests/functional/local.mk
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ nix_tests = \
flakes/show.sh \
impure-derivations.sh \
path-from-hash-part.sh \
store-import-export.sh \
path-info.sh \
toString-path.sh \
read-only-store.sh \
Expand Down
38 changes: 38 additions & 0 deletions tests/functional/store-import-export.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
source config.sh

clearStore

BUILT_STORE_PATHS=$(nix build -f ./dependencies.nix input1_drv input2_drv --no-link --print-out-paths | sort)

# Make sure that we require the `--format` argument.
expect 1 nix store export --recursive $BUILT_STORE_PATHS > "$TEST_ROOT/store-export" 2> /dev/null || \
fail "nix store export should require the --format argument"
nix store export --format binary --recursive $BUILT_STORE_PATHS > "$TEST_ROOT/store-export"

clearStore
IMPORTED_STORE_PATHS=$(nix store import < "$TEST_ROOT/store-export" | sort)

# Make sure that the paths we built previously are still valid.
for BUILT_STORE_PATH in $BUILT_STORE_PATHS; do
nix path-info "$BUILT_STORE_PATH" || \
fail "path $BUILT_STORE_PATH should have been imported but isn't valid"
done
# Make sure that all the imported paths are valid.
for IMPORTED_STORE_PATH in $IMPORTED_STORE_PATHS; do
nix path-info "$IMPORTED_STORE_PATH" ||
fail "path $BUILT_STORE_PATH should have been imported but isn't valid"
done

faketty () {
# Run a command in a pseudo-terminal.
script -qefc "$(printf "%q " "$@")" /dev/null
}

! faketty nix store export --format binary --recursive $BUILT_STORE_PATHS > /dev/null || \
fail "nix store export should refuse to write in a tty by default"
faketty nix store export --format binary --recursive $BUILT_STORE_PATHS --output-file - > /dev/null || \
fail "nix store export should accept to write in a tty if explicitly asked to"

nix store export --format binary --recursive $BUILT_STORE_PATHS --output-file "$TEST_ROOT/store-export2"
diff "$TEST_ROOT/store-export" "$TEST_ROOT/store-export2" || \
fail '`nix store export --output-file blah` should be equivalent to `nix store export > blah`'