Use non-zfs with persistent volumes and the proxmox CSI

[riddley]

Hello. I'm learning k8s and your blog post and repo have been invaluable. Thank you very much for both.

I'm trying to set up a cluster on a system that isn't using ZFS and it's almost entirely working other than persistent volumes. I changed talos/variables.tf to say datastore_id = optional(string, "local-lvm") and that seems to work great.

If I try to change bootstrap/volumes/persistent-volume/variables.tf and bootstrap/volumes/variables.tf in a similar fashion, I get the below error. I tried both local-lvm and just lvm, but it errors either way.

│ Error: unexpected response code '501': {"data":null,"message":"Method 'POST /api2/json/nodes/wu/storage/lvm/content/' not implemented"}
│
│ with module.volumes.module.proxmox-volume["duke"].restapi_object.proxmox-volume,
│ on bootstrap/volumes/proxmox-volume/config.tf line 5, in resource "restapi_object" "proxmox-volume":
│ 5: resource "restapi_object" "proxmox-volume" {

Also, I have to create the volume(s) at cluster creation time or, if I add it later, running tofu apply will shut down all of the vms for some reason. Thanks for any help you can provide.

[cmacnichol]

I believe you missed a few of the places storage is set. There should be 5 locations to change that in and 1 to change the img storage. Went through this same thing earlier today. Easiest thing to do is to pull the repo down and do a search of the files with vscode.

May put in a pull request to update the settings to use all variables, just not sure if it is possible with how it is designed currently as the modules don't like parent variables being passed in without a bit of work. Makes it hard to set the variables in one global location.

[sebiklamar]

Why changing a default value in variables.tf? The purpose of defining a variable (usually in variables.tf) is making a configuration setting variable, and here local-zfs is just a default value when the configuration setting is omitted because the variable is defined as optional. Hence, the right way is setting the optional configuration explicitely in the variable instantiation, here in main.tf.

You would need to add a datastore_id attribute for each of your nodes and a storage attribute to each of your volumes. For example:

  nodes = {
    "ctrl-00" = {
      host_node     = "abel"
      machine_type  = "controlplane"
      ip            = "192.168.1.100"
      mac_address   = "BC:24:11:2E:C8:00"
      vm_id         = 800
      cpu           = 8
      ram_dedicated = 28672
      igpu          = true
      # added variable attribute
      datastore_id  = "local-lvm"
    }
    "ctrl-01" = {
      ...
      datastore_id  = "local-lvm"
      ...

[cmacnichol]

@sebiklamar I do agree. I just think it is confusing having the customization variable values set in different places instead of just one location.

In this case I would think it would be preferrable to change the value to required and just define the value with a root variable in each of the node configs. This would enable it to inherit the value from a customization variable in the root variables file.

Not sure the best route here. I will say that when I first started working with this repo a few days ago, the hardest part was figuring out all the different variables and how they tied together in order to customize things to my environment. There has to be a better way to handle the "customization" variables so that they are all in one place for any new user to use.

[sebiklamar]

it is confusing having the customization variable values set in different places instead of just one location.

In terms of files, the one location is your main.tf (and not any variables.tf nor in any sub-modules' *.tf files which are encapsulating all the logic.

just define the value with a root variable in each of the node configs

Yes, in terms of DRY (Don't Repeat Yourself), defining the datastore_id only once in a root location could be beneficial. I say "could" and not "would" (be beneficial) because on the other hand-side that multi-level declaration of the nodes variable could make it complex for users for understanding and using the variable. In my opinion, having the datastore_id or any other variable on node level (e.g. the VLAN or CPU type) on node level allows different node configration, making the module more flexible.

What I've done in my main.tf (actually, in terragrunt.hcl) for pseudo-DRY is defining a local variable (outside of the nodes block) and then referring to it in the nodes code block. It's still repeating the same datastore_id = "${local.datastore_id}", so not really DRY, however you set the variable's value only once.

# please note that the following is terragrunt HCL language which might differ a bit from terraform/tofu
# also, it's my further development with additional variables
# hence, don't expect copy-paste working 100 % without adaptation
locals {
  env            = "${include.envcommon.locals.env}"
  root_path      = "${dirname(find_in_parent_folders())}"
  storage_vmid   = 9818
  vlan_id        = 108
  ctrl_cpu       = 2
  ctrl_disk_size = 10
  ctrl_ram       = 3072
  work_cpu       = 2
  work_disk_size = 10
  work_ram       = 3072
  cpu_type       = "x86-64-v2-AES"
  domain         = "home.iseja.net"
  datastore_id   = "local-enc"
}
...
  nodes = {
    "${local.env}-ctrl-01.${local.domain}" = {
      host_node     = "pve1"
      machine_type  = "controlplane"
      ip            = "10.7.8.181"
      vm_id         = 7008181
      cpu           = "${local.ctrl_cpu}"
      datastore_id  = "${local.datastore_id}"
      disk_size     = "${local.ctrl_disk_size}"
      ram_dedicated = "${local.ctrl_ram}"
      vlan_id       = "${local.vlan_id}"
      # update        = true
    }
    "${local.env}-ctrl-02.${local.domain}" = {
      host_node     = "pve4"
      machine_type  = "controlplane"
      ip            = "10.7.8.182"
      vm_id         = 7008182
      cpu           = "${local.ctrl_cpu}"
      datastore_id  = "${local.datastore_id}"
      disk_size     = "${local.ctrl_disk_size}"
      ram_dedicated = "${local.ctrl_ram}"
      vlan_id       = "${local.vlan_id}"
      # update        = true
    }

the hardest part was figuring out all the different variables and how they tied together in order to customize things to my environment

Yes, agree. The repo would benefit from more documentation. It's also subject of another thread (#143). That's where we as consumers could contribute something back to Vegard's excellent repo. He's, by the way, documenting already a lot on his blog.

I have documentation on my todo list because I not only want to contribute back to this repo, I also will need it for my further development of it (I've transformed Vergard's tofu/kubernetes code into a better re-usable terraform-proxmox-talos module). The thing is, as developer one often is focussing on the code and functionality and not on documentation. I've also fallen into the trap until now.

That said, I've moved one of the many documentation todo items into Ready state for getting picked up next.

Let's see who of us will be the first one with a PR ;-)

[riddley]

I'm "porting" his config into mine rather than using his directly and I don't and won't have zfs ever, so it seemed simpler to set the defaults for myself. This is the output of git grep local-zfs in his repo with the README/home-assistant output removed:

k8s/infra/storage/proxmox-csi/values.yaml:    storage: local-zfs
tofu/kubernetes/bootstrap/volumes/persistent-volume/variables.tf:    storage = optional(string, "local-zfs")
tofu/kubernetes/bootstrap/volumes/proxmox-volume/variables.tf:    storage = optional(string, "local-zfs")
tofu/kubernetes/bootstrap/volumes/variables.tf:      storage = optional(string, "local-zfs")
tofu/kubernetes/talos/variables.tf:    datastore_id = optional(string, "local-zfs")
tofu/kubernetes/variables.tf:#      storage = optional(string, "local-zfs")

and from mine:

tofu/k8s/bootstrap/volumes/proxmox-volume/variables.tf:    storage = optional(string, "lvm")
tofu/k8s/bootstrap/volumes/persistent-volume/variables.tf:    storage = optional(string, "lvm")
tofu/k8s/bootstrap/volumes/variables.tf:      storage = optional(string, "lvm")
tofu/k8s/talos/variables.tf:    datastore_id = optional(string, "local-lvm")

I haven't messed with the k8s/infra/storage/proxmox-csi part yet because I don't think it's referenced in the tofu code, is it?

I mostly opened this to find out about the error I'm getting. Does anyone have any advice about that?

[cmacnichol]

@sebiklamar I do like your implementation of using locals that way, may have to adopt that as well, it makes sense when you think in DRY terms. Feeling a bit behind now that I saw your repo with the project tracking, definitely seems like something I need as I keep bashing my head against different items at different times. I also really like the idea of the module you linked, still going over it as I have yet to get into terragrunt. May be time to change that.

@riddley That error looks like it is coming from the RestAPI call to create the volume on ProxMox. 501 usually means the command\content is not recognized. Looking at the RestApi code it and what your error says implies the error is here. That does not seem to match any of your storage names that you mentioned above. Do you have that set to LVM anywhere else?

/api2/json/nodes/wu/storage/lvm/content/'

Also, there is one call to the k8s data under the bootstrap function for values.yaml.

[riddley]

I've tried both local-lvm and just lvm and neither work. What is that string? Is it the name of the "Storage" found in the Proxmox instance? If so, local-lvm is the correct string. I tried lvm from reading the proxmox-csi docs, but I'll confess I didn't fully grok them.

Here's the version of the error with local-lvm instead:

╷
│ Error: unexpected response code '501': {"message":"Method 'POST /api2/json/nodes/wu/storage/local-lvm/content/' not implemented","data":null}
│
│ with module.volumes.module.proxmox-volume["duke"].restapi_object.proxmox-volume,
│ on bootstrap/volumes/proxmox-volume/config.tf line 5, in resource "restapi_object" "proxmox-volume":
│ 5: resource "restapi_object" "proxmox-volume" {

The only reference I can find to the k8s stuff is in the cilium section of main.tf.

[riddley]

I don't see an analog to this in the tf, could that be why? https://github.com/sergelogvinov/proxmox-csi-plugin/blob/main/docs/install.md#install-the-plugin-by-using-talos-machine-config

Edit: nope, not the problem.

[sebiklamar]

@riddley

I haven't messed with the k8s/infra/storage/proxmox-csi part yet because I don't think it's referenced in the tofu code, is it?

Multiple answers/facts here:

1. Indeed, you will need to set storageClass.storage in k8s/infra/storage/proxmox-csi/values.yaml to your datastore name as well.
2. Yes, the k8s values are not referenced from tofu code (other than cillium's values.yaml), hence initial cluster creation should work without errors regardless of k8s/.../proxmox-csi/values.yaml. Nevertheless, you will need a proper values.yaml setup once you instantiate proxmox-csi container after tofu cluster bootstrapping, i.e. when you fire up kubectl (e.g. kustomize build --enable-helm k8s/infra/storage/proxmox-csi | kubectl apply -f -)

Ah, you provided some more details while I was drafting my reply.

Here's the version of the error with local-lvm instead

Did you try (whatever) local/local-lvm in both configs (see item 2. above)? Both values should work as long as your datastore is enabled for "Disk image" type in Proxmox' Storage settings - which it is per default, AFAIK.

So, for going forward with your issue:

1. Could you please ensure proper datastore value setting in both main.tf and values.yaml and then tell us when the error message pops up (already after tofu or after k8s bootstrapping). I assume you already created the initial cluster (with tofu) successfully and then are only failing in instantiating proxmox-csi within k8s/ environment.
2. Did you do all prelimary steps in setting up proxmox-csi within Proxmox, e.g. user and role? See Vegard's blog post on proxmox-csi.
3. Did you succeed without any errors after tofu apply (for an intial cluster creation)?
4. What's the output of kustomize build --enable-helm k8s/infra/storage/proxmox-csi | kubectl apply -f -?

HTH -- Sebastian

PS: Once this is working... soon!
Another hint I got from Vegard's blog post: After proxmox-csi bootstrapping is succeeding, you can check for proper availability with

$ kubectl get csistoragecapacities -ocustom columns=CLASS:.storageClassName,AVAIL:.capacity,ZONE:.nodeTopology.matchLabels -A
k get -A pv`

proxmox-csi   821991632Ki   map[topology.kubernetes.io/region:homelab topology.kubernetes.io/zone:euclid]
proxmox-csi   916754956Ki   map[topology.kubernetes.io/region:homelab topology.kubernetes.io/zone:cantor]
proxmox-csi   960049560Ki   map[topology.kubernetes.io/region:homelab topology.kubernetes.io/zone:abel]

[riddley]

Thanks for your help!

In terms of 1, I'm not sure to which main.tf and values.yaml you refer, but I've definitely replaced all references to local-zfs with local-lvm. I'm able to bring up the cluster with no errors if I omit module "volumes" {...} from the top-level main.tf in the tofu/k8s directory.

For 2, I initially tried to do a manual setup of the user and role, but it complained about already existing when I tried to run tofu so I think this code does that initial setup https://github.com/vehagn/homelab/blob/main/tofu/kubernetes/bootstrap/proxmox-csi-plugin/config.tf

for 3, I am able to create the cluster w/ no errors if I omit the module "volumes" {...} from tofu/k8s/main.tf

for 4: I don't have kustomize installed and didn't know it was a requirement, so I'll get that installed.

Here's my output for your PS:

kubectl get csistoragecapacities -ocustom-columns=CLASS:.storageClassName,AVAIL:.capacity,ZONE:.nodeTopology.matchLabels -A
CLASS   AVAIL   ZONE