[WIP] async pickers take 2 using libuv uv_idle_t

[oberblastmeister]

note: This is a big work in progress. This is my second take to make telescope fully async. This time I am using coroutines and a coroutine scheduler that uses uv_idle_t. uv_idle_t runs something once on each iteration for the libuv event loop. This means we can kind of imitate threads and things will be fully async. It is kind of like telescope has it's own event loop. This is not usable yet as the highlighting doesn't work but otherwise there is great performance, this finally fixes the blocking on find files in my home directory! To fix the highlighting issue we can have a separate 'thread' running that only does highlights, so it doesn't flicker and will highlight correctly. Also, this still doesn't solve the problem of blocking on grep_string in my home directory. I think I have found that the bottleneck is the on_stdout function. The picker will lag when opening because of the sheer amount of entries and on_stdout calls. I am not sure how to address that as this pr is only for the lagging while typing, not the lagging when adding all the entries when opening the picker.

[tjdevries]

Have you checked out lua/telescope/algos/linked_list.lua? It's not exactly the same as what you're doing, but might be helpful? I'm not sure.

[elianiva]

This is so cool! It also introduce a weird sorting animation which I really like 😆
It sometimes still freezes but this is still a WIP so I guess that's expected.

[Conni2461]

I thought hey if @elianiva says its cool i give it a try. Home dir Telescope find_files hidden=true with stopwatch. One run on master and one run with that PR.
I went from ~ 25 sec to ~ 22 sec until i found my result and got a Preview. Its a improvement but not a big one.

And tbh i think we might wanna do something completely different for find_files. I could get to my file within ~ 18 sec with scandir, hidden = true obviously :) and that is not even the async one, so startup is super slow. But i think with a new finder that works with the scan_dir_async we might be even better than calling a external process. (And we are finally at the point where: "All lua, all the time." would apply) Just thinking out load. Wrote that code below in like 5 mins as proof of concept.

Edit: Just to clarify that script below ran on current master.

local finders = require('telescope.finders')
local make_entry = require('telescope.make_entry')
local pickers = require('telescope.pickers')
local conf = require('telescope.config').values

local scan = require'plenary.scandir'

local files = scan.scan_dir('/home/conni', { hidden = true })

local opts = {}

pickers.new(opts, {
  prompt_title = 'Files',
  finder = finders.new_table {
    results = files,
    entry_maker = make_entry.gen_from_file(opts),
  },
  sorter = conf.generic_sorter(opts),
  previewer = conf.file_previewer(opts),
}):find()

[oberblastmeister]

When I wrote this pr I wasn't thinking about raw speed. The fact that raw speed improves at all is great. I was more focused on the preemptibility. On master find_files will take a similar amount of time as this branch, however the user experience is bad. When I am typing nothing appears and there is no animation to show that it is processing, it just blocks completely. And then when I type new things on the prompt or delete it will still work on the old one and block. For example, I can type stuff onto the prompt and then click <C-W>. On master it will lag completely and it won't even show that I deleted the text. On this branch my text will be deleted and the picker will continue. This branch still has some work to do though such as highlighting better and less flickering.

[Conni2461]

Oh then i miss understood you :) yes this PR improves on that GJ :D

Edit: This can also easily be seen with my proof of concept. 😆 I should have done more testing

[oberblastmeister]

I also added a new feature. Since the coroutines can preempt so easily, I have made it so it stops working on the previous prompt when the prompt changes.

[tjdevries]

One thing we can do is also check the result of process_result or similar to determine if we should stop the execution.

[tjdevries]

You can put stuff like this in scratch/ if you want.

[oberblastmeister]

A plenary pr has superceded this one to be able to have better abstractions for async, then we can focus on using them for telescope

[tjdevries]

async pickers 3 and then asycn await stuff done now :)