Add kvm driver support

[jimmidyson]

Note that this requires cgo & dynamic linking to libvirt - this may be a deal
breaker but I'm happy to discuss if there is a way to get this in as I know of
a lot of users that would prefer kvm over virtualbox when running on Linux.

Fixes #238

[jimmidyson]

So it seems a bit easier... no need for cgo, no dynamic linking... libmachine checks PATH for drivers & executes them if found if available so all. I've reverted back to building with cgo disabled so this is still portable. This should work the same for any of the other supported plugins too which is pretty cool.

[dlorenc]

Hey,

I'm trying to follow along - it looks like you're still starting the driver in process in client.go, right?

How is the plugin binary itself getting compiled, are you relying on the one you have on your PATH from docker-machine?

[jimmidyson]

Well I'm not 100% sure how it works (yet) but it seems like the plugin architecture uses the plugin binary if available in PATH. It does not rely on docker-machine being in the PATH though. So the configuration of the plugin happens in-process & then libmachine looks through the PATH to see if the plugin binary is available & if so calls that appropriately with the specified configuration. A nice separation IMO & will allow minikube to support the full range of docker machine plugins more easily I guess.

[dlorenc]

Cool, I like the overall approach but have a few comments that might make this more difficult ;)
My understanding of the driver model is sort of like this:

• The minikube binary first starts up in non-driver mode. It acts as a normal CLI, parses flags and decides which driver to use.
• When you go to create that driver, libmachine will internally start the driver process, then send paramters to it over an RPC mechanism. This has two modes:
  • Use the same binary (minikube)
  • Use a different binary (libmachine will look on your path for a binary with the right name)

We currently operate in mode 1, where minikube acts both as the CLI and as the driver. libmachine sets some environment variables to tell the binary it calls (in this case minikube) what to act as. The code in minikube that checks what to act as is here: https://github.com/kubernetes/minikube/blob/master/pkg/minikube/machine/client.go

So with your PR, I'm unclear on how this could be using the standalone binary. If this boolean is set to True:

minikube/pkg/minikube/machine/client.go

Line 43 in 905afe9

localbinary.CurrentBinaryIsDockerMachine = true

I think libmachine will always use "minikube" as the driver process:

minikube/vendor/github.com/docker/machine/libmachine/drivers/plugin/localbinary/plugin.go

Line 102 in d74ce77

if CurrentBinaryIsDockerMachine {

[jimmidyson]

At that makes things clearer. Reading the code what you describe is only applicable to core drivers, not drivers from plugins. Relevant line is

minikube/vendor/github.com/docker/machine/libmachine/drivers/plugin/localbinary/plugin.go

Line 100 in d74ce77

for _, coreDriver := range CoreDrivers {

. KVM is provided by a plugin, hence the fork/RPC.

[dlorenc]

Ah ok, I didn't realize there was that distinction. In that case, we probably don't need this change here: https://github.com/kubernetes/minikube/pull/242/files#diff-b9679bde9fd46c8d4f93dd0739ad6d01R39

since minikube will never act as the KVM driver.

If you remove that, then I don't think any of the new dependencies need to get vendored in, since all that is packaged up in docker-machine-kvm, right?

[luxas]

Re cgo linking, it is not an issue to include c code if one knows how to compile statically.
See kubernetes/kubernetes#26863 for more information

Maybe the standalone binary solution is better, because @dlorenc mentioned earlier that minikube shouldn't be a direct fork of docker-machine for kubernetes with all the drivers compiled in.
IMO, the "bring your own docker-machine driver" solution is better.

[jimmidyson]

Maybe the standalone binary solution is better, because @dlorenc mentioned earlier that minikube shouldn't be a direct fork of docker-machine for kubernetes with all the drivers compiled in.
IMO, the "bring your own docker-machine driver" solution is better.

Agreed - I'll refactor this to try to get closer to that. Thanks for the feedback!

[jimmidyson]

So now this has no new dependencies & only requires docker-machine-driver-kvm to be in the PATH to work properly. The one small caveat is that the KVM specific driver options struct has been copied into minikube codebase which could potentially cause compatibility issues if docker-machine-driver-kvm changes options breaking compatibility. This approach should also work nicely for any other plugin drivers, especially tihinking about xhyve for osx users.

[dlorenc]

Can you document how to install the docker-machine-kvm driver? It looks like they publish versioned releases, so it would probably be good to include what version you've tested this with as well.

[jimmidyson]

Done.

[dlorenc]

Nice! This looks ready to go. One small request in the docs, then LGTM. I'm excited to try this out!

[dlorenc]

Also you'll have to run "go run gen_help_text.go" to update the help text so tests pass.

[jimmidyson]

Hopefully should be good to go now.

[dlorenc]

Hey,

I pulled your branch to try it out, and I'm having some trouble. I'm getting this error:

(minikubeVM) Failed to create the VM: [Code-67] [Domain-20] unsupported configuration: unknown disk cache mode ''

Any idea what I'm doing wrong? I installed libvirt and qemu-kvm, and I'm on an Ubuntu system.

[jimmidyson]

D'oh! I tested with a local build which didn't have those options - fixed & tested against 0.7.0 (definitely!) now.

[dlorenc]

Nice, I got it working!

[jimmidyson]

Yay!