-
Notifications
You must be signed in to change notification settings - Fork 35
Description
I've attempted to use Filter in two real world examples now but for one I had compromise on my stats reporting and for the other I had to back out of using it. I need to report the # of original inputs and the # post filtering which doesn't seem to be easily possible. I can use Scan or some other side-effect support operation, but then I need to make the side effect work across multiple machines which is annoying and expensive for something as simple as a count. Google's mapreduce offered counters which could be used for this task as this; this mechanism though seems overly general and was extensively abused so I'm not necessarily advocating for it. Given bigslice's ability to carry results through the graph via the slices themselves, maybe it makes sense to add stats to the core slice structure to report on progress through the graph - the simplest being # invocations per operation, but more detailed/extensive ones also being possible.