df.show() would cause multi runs of operations

[xy-xin]

Suppose we have two UDFs: udf1 and udf2, and the following code:

df = xxx

df = df.with_column("output1", udf1(col("input1")))
**df.show()**

df = df.with_column("output2", udf2(col("input2")))
**df.show()**

In this case, udf1 will be triggered twice, which is not what the user expects.

We can work around this by using collect():

df = xxx

df = df.with_column("output1", udf1(col("input1"))
**df.collect()**
**df.show()**

df = df.with_column("output2", udf2(col("input2"))
**df.show()**

However, this requires the user to have experience with how Daft works, which can be challenging for algorithm developers.

In conclusion, users must understand how Daft works and what happens when calling show(). Otherwise, this behavior may confuse users.

[srilman]

I think the misunderstanding here is that df.show() does more than just run df. It also applies a limit to the end of the query plan.

So if df represents a plan like

ParquetScan -> Projection -> UDF

then df.show will actually run

ParquetScan -> Limit -> Project -> UDF

The limit it in the front because of limit pushdown. And so it is not always possible to modify df and run df.show() again and expect it to do the same thing. For example, if we added a groupby, we can't push the limit all the way though and the plan is now too different.

In your example, it is probably fine, but we need to somehow save the plan and showed output together, which we don't do for memory purposes. Wdyt?

[xy-xin]

Thanks for your reply, @srilman

I understand and agree with your point. However, what I wanted to highlight is that in some cases, calling df.show() twice may trigger two executions of the UDF, especially if the UDF has side effects. Algorithm developers do not have much experience on how Daft (or other SQL-like engines) works internally, and this behavior can be confusing for them.

If we don’t have a good solution to this issue and decide to address it through better documentation and user guidance instead, I’m fine with that approach as well.