Tuning with filters recalculates filter values each iteration

[Prometheus77]

Not a bug, but a recommendation to speed up execution. Currently, when using a filter to tune (e.g. tuning on fw.perc), it recalculates the filter values at each tuning iteration, which in the case of the default filter method (randomForestRFSRC) can be quite time consuming. Would be more efficient to have the tuner check whether there is a filter wrapper in the learner, and if so, to calculate the filter values at the beginning of the tuning process and then simply pass them forward into the filter at each iteration.

A prerequisite for this would be the ability to pass pre-defined filter values into a filter, which I do not believe is currently possible.

[mazgit82]

I wonder if it would be possible to define another kind of hyperparameter that must be used at predict time. For instance, this could be useful for methods such as gbm, for which you can do something like preds = predict(model, n.trees = 100) even if the model contains 1000 trees. This way, you do not have to build a 100-tree forest and another 1000-tree forest.

I believe this could also be useful for the filters you mention.

[mb706]

(Commenting on the issue as named in the title)

The recalculation of filter values has annoyed me at times as well, and I think it should be accelerated. It would probably be possible to cache the results of generateFilterValuesData, globally or with a cache parameter given to that function (and managed by the FilterWrapper / the Filter CPO).

The problem with caching is that some feature filters are nondeterministic and should not be cached.

There are two R libraries that perform caching that I'm aware of: R.cache and memoise. The latter seems to be more lightweight since R.cache apparently doesn't support in-memory caching (?). A potential problem with all of these is that they hash the whole Task data--would probably be fast relative to the filter value creation (most filter value functions need to sweep the whole task as well), but might be slow in absolute terms.

An idea that might be useful in other places as well is to have the Task store its own hash vale. Since Tasks already contain an env, a getTaskHash() function might calculate the hash and store it in this environment to avoid needing to re-calculate it.

[pat-s]

I'll tackle this issue in the near future. Together with the FSelectorRcpp implementation we should get a significant speed up!

[mb706]

Maybe my implementation of cached preprocessing can give you some ideas.

[pat-s]

Got a working solution. Tuning fw.perc on 5 random search iterations with the same filter method.

# with caching
[Tune-x] 1: fw.perc=0.962; C=4.08; sigma=1.23
[Tune-y] 1: mse.test.mean=0.8345549; time: 0.1 min
[Tune-x] 2: fw.perc=0.403; C=195; sigma=0.152
[Tune-y] 2: mse.test.mean=0.8345549; time: 0.0 min
[Tune-x] 3: fw.perc=0.288; C=0.0104; sigma=0.0106
[Tune-y] 3: mse.test.mean=0.8373343; time: 0.0 min
[Tune-x] 4: fw.perc=0.482; C=0.0326; sigma=0.0196
[Tune-y] 4: mse.test.mean=0.8375375; time: 0.0 min
[Tune-x] 5: fw.perc=0.674; C=0.00189; sigma=16.2
[Tune-y] 5: mse.test.mean=0.8373490; time: 0.0 min
[Tune] Result: fw.perc=0.962; C=4.08; sigma=1.23 : mse.test.mean=0.8345549

Unit: seconds
 min       lq     mean   median       uq      max neval
7.704957 7.704957 7.704957 7.704957 7.704957 7.704957     1

# without caching
[Tune-x] 1: fw.perc=0.962; C=4.08; sigma=1.23
[Tune-y] 1: mse.test.mean=0.8345549; time: 0.1 min
[Tune-x] 2: fw.perc=0.403; C=195; sigma=0.152
[Tune-y] 2: mse.test.mean=0.8345549; time: 0.1 min
[Tune-x] 3: fw.perc=0.288; C=0.0104; sigma=0.0106
[Tune-y] 3: mse.test.mean=0.8373343; time: 0.1 min
[Tune-x] 4: fw.perc=0.482; C=0.0326; sigma=0.0196
[Tune-y] 4: mse.test.mean=0.8375375; time: 0.1 min
[Tune-x] 5: fw.perc=0.674; C=0.00189; sigma=16.2
[Tune-y] 5: mse.test.mean=0.8373490; time: 0.1 min
[Tune] Result: fw.perc=0.962; C=4.08; sigma=1.23 : mse.test.mean=0.8345549

Unit: seconds
 min       lq     mean   median       uq      max neval
24.66161 24.66161 24.66161 24.66161 24.66161 24.66161     1