Support "Feature Preview" rollout of settings/defaults.

[JacksonKearl]

A few times now we've done "Phased Rollouts" of new feature (welcome, native notebooks, etc.), default setting values. It'd be nice to have some sort of standardized way both for

• VS Code core to have phased roll-out of a change in default setting values (including new feature flags)
• Users to have a way to view a list of the "experimental" changes currently available/enabled, ideally with a link to a tracking issue on GH

Perhaps an @experiments tag in settings editor could help here? Or something new, maybe taking inspi from GitHub's "Feature Preview" modal.

cc @roblourens @digitarald @luabud

[sbatten]

Some thoughts on this from the experiment service (now called workbench assignment service) side...

@JacksonKearl had already made all the technical changes to override experiment values (explicitly on or off).

The configuration service cannot rely on the assignment because it needs to rely on settings for overrides, so the settings editor should do the merging logic. Right now, overrides are hidden settings with fixed names based on the treatment variables. I would expect this could be done by adding explicit settings that would allow you to describe the experiment. The settings editor would then need to handle the different states (am i opted in or out by choice or by the treatment group).

[digitarald]

Love this for many reasons, as it naturally allows the rollout both via outreach leading to manual opt-in and experiments leading to opting in a random set of users.

To support this for extensions, I'd love to see this '@experiment' tag supported in settings contributed by package.json as well.

[sandy081]

@JacksonKearl @sbatten Can you please shed some light here - what is expected from configuration land?