Using setDF() within a function will not modify global object, while setDT() does?

[MatthieuStigler]

Note: this was first asked on stack, but asking here as it seems highly technical?

I am converting back and forth from data.frame to data.table using a workflow like: setDT() + some data.table operation + setDF() (see this post). Using setDF() will correctly transform back my input into a data.frame.

However, when I use exactly the same code, but within a function, the input is not transformed back into an data.frame anymore after a setDF() call, while the output will still correctly be converted to a df. I guess this comes from am ambiguity at that point whether the object of setDF() refers to the local data within the function or the global data in the user workspace? Strangely, this does not happen to setDT()?

So how can I ensure that I can use the workflow setDT() + some data.table operation + setDF() within a function, and that setDF() modifies back the input to that function?

library(data.table)
data(iris)

## using setDF outside of function
setDT(iris)
iris[, Sepal.Length.Mean := mean(Sepal.Length), by = Species]
setDF(iris)

class(iris)
#> [1] "data.frame"

## using setDF within a function
fo <- function(df){
  setDT(df)
  df[, Sepal.Length.Mean := mean(Sepal.Length), by = Species]
  setDF(df)
  df
}

res <- fo(iris)
class(iris)
#> [1] "data.table" "data.frame"

^{Created on 2020-11-17 by the reprex package (v0.3.0)}

[ColeMiller1]

#4589 has discussion related to this. There are also links to stack overflow which are good references.

My oversimplified summary is that the global object and function object diverge in address after the setDT(df) calls. So while the first setDT call affects both objects, the setDF() only affects the copy in the function environment.

A solution is to use dt = as.data.table(df) within your function to ensure you are working on a copy at the very beginning.

[ecoRoland2]

@ColeMiller1 I think OP wants to avoid the copy. It's not at all obvious why a deep copy occurs within fo. If your summary is correct, it should be added to help("setDT").

[ColeMiller1]

@ecoRoland2 Yes, I agree that it is confusing and unexpected behavior. I should also note that this is just my understanding of the other issues. I don't have a complete understanding of why it is happening nor if my summary is correct.

I also think this is a duplicate of the other issues.

Regardless, to answer the question:

So how can I ensure that I can use the workflow setDT() + some data.table operation + setDF() within a function, and that setDF() modifies back the input to that function?

It does not seem to be possible without a deep copy. In addition to documentation, it seems like we should make setDT() force copies if done at a higher frame. The most conservative would be to force a copy outside of sys.nframe() > 1. e.g.,

setDT2 = function(dt, frame_copy = 1L) {
    if (sys.nframe() > frame_copy) {
      setDT(copy(dt))
      print("copy made :(")
    }
     else {
       setDT(dt)
        print("no copy; yes!")
     }
}

dummy_fx = function(dt){
  setDT2(dt)
}

library(data.table)
dt = as.data.table(iris)

setDT2(dt)
#> [1] "no copy; yes!"
dummy_fx(dt)
#> [1] "copy made :("

This would probably be a bad change. I used reprex::reprex() but had to alter the output because there are a lot of functions called before getting to setDT2(dt).

[MatthieuStigler]

Thanks for both comments!

Alternatively, instead of having setDT() to not change the input in the user workspace when called form within a function, is there a way to have setDF() change an object in the workspace from within a function?

Thanks!

[ecoRoland2]

@MatthieuStigler You are asking the wrong question. The whole point of the setDT/setDF combo is to avoid a deep copy. However, it doesn't seem like a deep copy can currently be avoided if you deal with a data.frame passed as a function parameter. Thus, it would be best to start with a deep copy and avoid modifying iris in the first place:

fo <- function(df){
  df <- as.data.table(df)
  df[, Sepal.Length.Mean := mean(Sepal.Length), by = Species]
  setDF(df)
  df
}

[jangorecki]

@ecoRoland2 your example seems irrelevant, there is no unexpected behavior in it.

fo <- function(df){
  df <- as.data.table(df)
  ## we made a copy with as.data.table, check new address
  print(address(df))
  df[, Sepal.Length.Mean := mean(Sepal.Length), by = Species]
  setDF(df) ## df points to an object created within this function
  df
}
df = fo(iris)
#[1] "0x56278d3f34a0"
print(address(df))
#[1] "0x56278d3f34a0"
class(df)
#[1] "data.frame"

My guess is that setDF is less robust than setDT because it was less frequently used. And #4589 is not related to this one. And AFAIR there was no issue reported like this before, so it is not a duplicate.
But one should look into the source of setDT vs setDF to ensure about my guess.

[ecoRoland2]

@jangorecki I know there is nothing unexpected in this. That's why I suggest OP uses that approach (for now).

[ColeMiller1]

From the other thread which is very similar to this issue:

I was hoping to gradually migrate to data.table, by pinpointing performance data.frame bottlenecks, locally converting them to data.table (without copies!), perform optimized data.table operations and convert back.

The unexpected part is that within the scope of a function, the set* operators affect the global object but subsequent set* operators only affect the object in the scope of the function if a copy has been made. I am not sure setDF() is the culprit.

I compiled and tested using @OfekShilon's suggestion (by way of GKi's investigation) in #4784 of switching these lines:

data.table/R/data.table.R

Lines 2723 to 2724 in 8480b6a

	setattr(x, "class", .resetclass(x, 'data.frame'))
	setalloccol(x)

No errors and it fixes the issue in this thread. It seems like the only undesired behavior is that a function will update the caller object if the same column name is modified in place. But that is also true of 1.13.2.

## data.table package modified to switch lines in setDT()
df <- data.frame(a=c(1,2), b=c(3, 4))

foo1 <- function(z) { 
  setDT(z)
  z[, `:=`(b = 5, c = 6)]
  # z[, b:=5]   # will apply to the caller copy
  # z[, c:= 6]  ## will not apply
  data.table::setDF(z)
}

foo1(df)
class(df)
## [1] "data.frame"
df
## a b
## 1 1 5
## 2 2 5

I also looked at memory. Both 1.13.2 and the modified 1.13.3 used the same amount of memory according to bench::mark(). So, should we look at switching those two lines, @jangorecki ?

[OfekShilon]

@ColeMiller1 Please note that this originally suggested fix looks like it causes magrittr in-pipe (%>%) to stop working with setDT:

df<-data.frame(a=1)
df %>% fixedSetDT
class(df)  # "data.frame" !  

The reason seems to be that this fix exposes a different bug.
In df %>% setDT, df changed its class only by the 'class leakage to caller' bug that the suggested line-switch fixes. When you fix it, df no longer changes class.

In reality df %>% setDT never worked as intended. It did change df's class (as a side effect of this line-switch bug), but nothing more. For example:

d <- data.frame(a=1:2)
setDT(d)
truelength(d)    # 1025
d <- data.frame(a=1:2)
d %>% setDT
truelength(d)   # 0  !

So maybe it's worth fixing the %>% interaction before this fix, to avoid breaking existing code.

[ColeMiller1]

So maybe it's worth fixing the %>% interaction before this fix, to avoid breaking existing code.

@OfekShilon My understanding is that df%>%setDT does not work now; I'm not sure how we would break existing code if its already broken.

Note, I think that the magrittr pipes are somewhat different for the set*() functions. That is, the pipe is setting the lhs object as a data.table. The lhs object is not really the original named object, but instead a shallow copy referencing the original object AFAIU. I believe your recommended code change would actually be expected behavior based on the other issues you have raised; df %>% setDT would result in the original df object to still be a data.frame.

Stylistically, if pipes are being used, dtplyr is a pretty good option.

[OfekShilon]

@OfekShilon My understanding is that df%>%setDT does not work now; I'm not sure how we would break existing code if its already broken.

@ColeMiller1 I'm sorry I wasn't clear, let me try again: the suggested fix (line switch) does change the semantics of code and when tested on our code base did break things.

df <- data.frame(a=1:2)
df %>% setDT
class(df)  # "data.table  data.frame"

df <- data.frame(a=1:2)
df %>% fixedSetDT
class(df)  # "data.frame"

True, the first df was a "half baked" data.table (with no attribute properly set except its class) but it was marked as data.table and method dispatch in the following code directed execution to data.table methods. This changes if you switch the lines under discussion.

Note, I think that the magrittr pipes are somewhat different for the set*() functions. That is, the pipe is setting the lhs object as a data.table. The lhs object is not really the original named object, but instead a shallow copy referencing the original object AFAIU. I believe your recommended code change would actually be expected behavior based on the other issues you have raised; df %>% setDT would result in the original df object to still be a data.frame.

Personally I can live with this behavior, but it certainly isn't the current one:

df<-data.frame(a=1:2)
df %>% setnames("c")
names(df)    # "c"

Stylistically, if pipes are being used, dtplyr is a pretty good option.

Thanks! I wasn't aware of this package, will check.

[OfekShilon]

@ColeMiller1 Sorry, I take back my comment. Two wrongs don't make a right - it's better to fix this class-leakage-to-caller from setDT, and deal with magrittr pipes separately.