RFE: support a global config file

[scop]

Would be nice to see lychee add support for a global (per user) config file. A good location for it would be ~/.config/lychee.toml.

Based on my short experience with lychee, I find myself hitting a lot of rate limits. I also do one-off runs against various projects, and having to configure lychee for each of them would be a chore. On the other hand remembering the command line options needed is also one, and e.g. for GITHUB_TOKEN, I don't want that in my global env as it might end up being used inadvertently by something.

So a per user config would nicely fit the bill. It could for example be used as the basis, merged with the per project one if found, with the latter having precedence.

A shell alias with the desired options written in it serves as a workaround meanwhile.

[mre]

To future-proof it a bit, I would put the global config into a separate folder, e.g. ~/.config/lychee/lychee.toml. Then we could add more files (like a global .lycheeignore file) there as well.

But that assumes we'd want to have a global config file in the first place. 😉

As for the rate-limits, this should significantly be improved once #1605 lands. We've put a lot of effort into per-host rate-limiting and configuration. In our tests, we found that we ran into significantly fewer rate limits. If possible, could you take a look at the pull request and try it locally (without any additional config)?

cargo install --git https://github.com/lycheeverse/lychee --branch per-host-rate-limiting

As for the GitHub token, the argument could be made that adding it to a global config file is even worse. Attackers often scan known locations for tokens and they could add the lychee config to the list of well-known paths. I agree that a global environment variable is not ideal, but arguably safer, as it's ephemeral at least.

The alternative is to use context-based environment variable tools, such as direnv, which automatically loads environment variables based on the current directory. You can create a .envrc file in your project root with export GITHUB_TOKEN=your_token_here, and direnv will load it when you enter the directory and unload it when you leave.

Tools like 1Password also allow you to handle these secrets safely. See here. There are many other tools and solutions such as zsh-autoenv for many shells.

The key advantage of these approaches is that they provide token access only when and where needed, reducing the attack surface compared to global environment variables or config files. They also make it easier to use different tokens for different projects or contexts, which would be hard to do with a global config file.

I'm generally not against a global config file, but combining per-project .envrc files with the upcoming per-host rate-limiting improvements should provide both better security and performance.

[scop]

Thanks for considering this.

I use mise to manage per project environment that activates on entry etc and have used other tools to the same effect for a long time. But that's not a replacement for this, I'd still need to set up something for each dir I want this to happen. That's a step I'd like to avoid for one-off runs against things I may clone in /tmp once, fix up links and submit a PR, and maybe never revisit again. It's not a problem for projects whose working dir I'll keep around.

Global env vars are very likely found in easily guessable ~/.bashrc or similar, so from security PoV there's no difference between that or tool specific default config. My point regarding this is orthogonal -- I want to have some control over what apps are using which tokens, and if there's a global unnamespaced GITHUB_TOKEN, I've no idea what might pick it up.

Rate limits and the token are just two things I happened to immediately run into. It's great to hear that the OOTB unconfigured experience might get better re the former, but there's a lot of things might want to configure in a global config files, in more permanent setups rather than the mentioned /tmp one-off corner case. These are typically things that are specific to my setup and not appropriate for let's say a project specific config from where it would be imposed on all users. The project might want to have some config that applies to everyone and in that case the config would be in git, and without a global config file, there's no place for me to add my settings (putting it in the project one would mark it annoyingly dirty and risk me inadvertently committing and pushing it).

Just skimming the available config options, ones I might want to set for myself only in addition to the mentioned ones include cookie-jar, threads, timeout, cache, mode, format, offline, basic_auth etc etc. These are not that appropriate for a shared project wide config. Ones good for that use include exclude, include, extensions, exclude_path, base_url and so on.