List Casting Explained

1 Introduction

Hierarchical data is surprisingly common, and are commonly represented in by nested lists.

Broadcasted operations can be performed over dimensions, but not through nesting or hierarchies.
Therefore, it is useful to be able to cast nested lists into dimensional lists.
The ‘broadcast’ package provides the cast_hier2dim() to cast a nested list into a dimensional list (AKA a recursive array).

Casting between nested and dimensional lists is not only useful for broadcasting, however.
Casting nested lists to dimensional lists has its own merits, as dimensional lists have some advantages over nested lists beside the broadcasting, such as the following:

• Performing sub-set operations on multiple recursive subsets (using the [[ and [[<- operators) requires a (potentially slow) loop, whereas multi-dimensional subsets (using operator forms like [..., ...] and [..., ...]<-) are vectorized and generally much faster.
• Re-organizing dimensions of a recursive array is generally much easier, faster, and more straight-forward than re-organizing hierarchies of a nested list.

This Vignette gives an overview of the functions ‘broadcast’ provides to cast between nested and dimensional lists.

 

2 Cast Hierarchical List to Dimensional List

2.1 Introduction

The cast_hier2dim() function casts a nested list into a dimensional list.
This section gently introduces the properties of this function through a series of examples, where each subsequent example builds on the previous one.
Familiarity with nested lists and dimensional lists (i.e. arrays of type list) is essential to follow these examples.

 

2.2 Example 1: Basics

For a first example, consider the following list:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

Before actually casting x into a dimensional list, one may want to know what the dimensions will become when casted as a dimensional list;
The hier2dim() function shows you that:

hier2dim(x)
#>       
#> 3 2 2

It returns the dimensions c(3, 2, 2).

Let’s now cast x as a dimensional list:

x2 <- cast_hier2dim(x) # actually cast nested list into dimensional list
print(x2)
#> , , 1
#> 
#>      [,1]         [,2]        
#> [1,] numeric,10   numeric,10  
#> [2,] numeric,10   numeric,10  
#> [3,] character,10 character,10
#> 
#> , , 2
#> 
#>      [,1]         [,2]        
#> [1,] numeric,10   numeric,10  
#> [2,] numeric,10   numeric,10  
#> [3,] character,10 character,10

Using the default arguments, element x[[i]][[j]][[k]] corresponds to element x2[k, j, i] (for all i, j, and k).
This can be changed, as will be shown in a later example.

As shown in the results above, cast_hier2dim() will obviously not preserve names by default.
It is trivially easy to set the dimnames of x2, using hiernames2dimnames():

dimnames(x2) <- hiernames2dimnames(x)
print(x2)
#> , , group1
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

There, the names are now correct.

As shown above, will display a dimensional list more compactly than a nested list.
Depending on the situation this may be either be desirable or undesirable.

One can print x2 less compactly without much effort by flattening it, using the cast_dim2flat() function.
We only need to see a portion of the list in detail, so let’s look at class1 from group 1 in the flattened form:

cast_dim2flat(x2[, 1, "group1", drop = FALSE])
#> $`['height', 'class1', 'group1']`
#>  [1] 170.6128 169.1722 169.5070 169.2417 171.1591 169.9541 167.2718 171.3536
#>  [9] 169.7714 169.9673
#> 
#> $`['weight', 'class1', 'group1']`
#>  [1] 78.89870 79.76817 79.09018 82.11039 79.99375 78.33916 79.79592 78.71359
#>  [9] 80.11941 79.15838
#> 
#> $`['sex', 'class1', 'group1']`
#>  [1] "F" "F" "M" NA  "M" "F" "F" "M" "M" "F"

 

Dimensional lists can be easier to work with than hierarchical lists.
Consider, for example, printing the height of the first class of every group in a list - let’s compare how to do this in a nested list vs a dimensional list.

With a nested list, doing this takes a slow, messy for-loop:

for(i in seq_along(x)) {
  print(names(x)[i])
  x[[i]][[1]][["height"]] |> print() # slow for-loop, messy code
}
#> [1] "group1"
#>  [1] 170.6128 169.1722 169.5070 169.2417 171.1591 169.9541 167.2718 171.3536
#>  [9] 169.7714 169.9673
#> [1] "group2"
#>  [1] 169.0435 171.5157 171.0296 169.0597 168.5510 168.6701 168.9254 170.2627
#>  [9] 170.3598 170.0228

With a dimensional list, the very same thing can be done with sleek, vectorized code; no messy loop needed:

x2["height", 1L, ] |> print()
#> $group1
#>  [1] 170.6128 169.1722 169.5070 169.2417 171.1591 169.9541 167.2718 171.3536
#>  [9] 169.7714 169.9673
#> 
#> $group2
#>  [1] 169.0435 171.5157 171.0296 169.0597 168.5510 168.6701 168.9254 170.2627
#>  [9] 170.3598 170.0228

x2["height", 1L, , drop = FALSE] |> cast_dim2flat() # same but more informative
#> $`['height', 'class1', 'group1']`
#>  [1] 170.6128 169.1722 169.5070 169.2417 171.1591 169.9541 167.2718 171.3536
#>  [9] 169.7714 169.9673
#> 
#> $`['height', 'class1', 'group2']`
#>  [1] 169.0435 171.5157 171.0296 169.0597 168.5510 168.6701 168.9254 170.2627
#>  [9] 170.3598 170.0228

It is also easier to re-arrange dimensions - for example using aperm() - than it is to re-arrange hierarchies.

 

2.3 Example 2: Cast from outside to inside

In Example 1, the default arguments were used for cast_hier2dim().
One of these arguments is in2out, which defaults to TRUE.

Consider a nested list x with a depth of 3, and a dimensional list X2 with 3 dimensions, where the relationship between x and x2 can be expressed as x2 <- cast_hier2dim(x, ...).
Given this, the following can be stated about in2out:

• If in2out = TRUE, which is the default and used in Example 1, element x[[i]][[j]][[k]] corresponds to element x2[k, j, i] (for all i, j, and k).
  
• If in2out = FALSE, element x[[i]][[j]][[k]] corresponds to element x2[i, j, k] (for all i, j, and k).

The default of in2out = TRUE was chosen, because elements in subsequent rows are close to each other, while elements in subsequent layers (third dimension) are generally not close to each other, and the default of in2out = TRUE attempts to retain that behaviour.

For this example, the same list will be used as in Example 1:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

Let’s once again cast this list to a dimensional list, but this time use in2out = FALSE:

hier2dim(x, in2out = FALSE) # check once again the dimensions
#>       
#> 2 2 3
x2 <- cast_hier2dim(x, in2out = FALSE) # actually cast nested list into dimensional list
print(x2)
#> , , 1
#> 
#>      [,1]       [,2]      
#> [1,] numeric,10 numeric,10
#> [2,] numeric,10 numeric,10
#> 
#> , , 2
#> 
#>      [,1]       [,2]      
#> [1,] numeric,10 numeric,10
#> [2,] numeric,10 numeric,10
#> 
#> , , 3
#> 
#>      [,1]         [,2]        
#> [1,] character,10 character,10
#> [2,] character,10 character,10

x2 is the casted list. Since in2out = FALSE, element x[[i]][[j]][[k]] corresponds to element x2[i, j, k] (for all i, j, and k).

Once again it is trivially easy to set the dimnames of x2, using hiernames2dimnames() (available from version 0.1.5):

# this time, in2out = FALSE
# so we go from the surface names to the deepest names
dimnames(x2) <- hiernames2dimnames(x, in2out = FALSE)
print(x2)
#> , , height
#> 
#>        class1     class2    
#> group1 numeric,10 numeric,10
#> group2 numeric,10 numeric,10
#> 
#> , , weight
#> 
#>        class1     class2    
#> group1 numeric,10 numeric,10
#> group2 numeric,10 numeric,10
#> 
#> , , sex
#> 
#>        class1       class2      
#> group1 character,10 character,10
#> group2 character,10 character,10

There, the names are now correct.

One can print x2 less compactly without much effort by flattening it, again using the cast_dim2flat() function.
We only need to see a portion of the list in detail, so let’s look at class1 from group 1 in the flattened form:

cast_dim2flat(x2["group1", 1, , drop = FALSE])
#> $`['group1', 'class1', 'height']`
#>  [1] 171.0168 170.0979 168.3799 170.5498 170.7448 171.3180 170.5342 169.5416
#>  [9] 169.8338 171.2166
#> 
#> $`['group1', 'class1', 'weight']`
#>  [1] 81.07829 78.51196 80.42874 80.36066 79.86497 80.52181 81.83435 80.33834
#>  [9] 78.91182 77.89423
#> 
#> $`['group1', 'class1', 'sex']`
#>  [1] "M" "F" "M" "M" NA  "F" "F" NA  "F" NA

 

2.4 Example 3: Padding

For Example 3, we take the same list as before, but remove x$group1$class2:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

Let’s first check what dimensions it will get when casted using hier2dim():

hier2dim(x)
#>         padding         
#>       3       2       2

The dimensions are the same as in Example 1: c(3, 2, 2).
But notice the names of the output are different: the second element has the name “padding”; this indicates that some columns won’t have enough elements to completely fill the column, and so additional elements will be added as padding.

So let’s cast this list as dimensional:

x2 <- cast_hier2dim(x)
print(x2)
#> , , 1
#> 
#>      [,1]         [,2]
#> [1,] numeric,10   NULL
#> [2,] numeric,10   NULL
#> [3,] character,10 NULL
#> 
#> , , 2
#> 
#>      [,1]         [,2]        
#> [1,] numeric,10   numeric,10  
#> [2,] numeric,10   numeric,10  
#> [3,] character,10 character,10

Subset x2[, 2, 1] is filled with NULL; this is the place where x$group1$class2 was in Example 1, but since it’s not there, we need to fill something.

To make it make obvious, let’s give the array proper dimnames:

dimnames(x2) <- hiernames2dimnames(x)
print(x2)
#> , , group1
#> 
#>        class1       class2
#> height numeric,10   NULL  
#> weight numeric,10   NULL  
#> sex    character,10 NULL  
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

Again, element “class2” is missing from element “group1”, but not from “group2”, and so padded with NULL when the list is casted as dimensional.

Sometimes, a different value than NULL is desired for padding.
So let’s replace the padding value with something really obvious, using the padding argument:

x2 <- cast_hier2dim(x, padding = list(~ "this is padding!"))
dimnames(x2) <- hiernames2dimnames(x)
print(x2)
#> , , group1
#> 
#>        class1       class2             
#> height numeric,10   ~"this is padding!"
#> weight numeric,10   ~"this is padding!"
#> sex    character,10 ~"this is padding!"
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

Once again, one can print or present x2 less compactly by flattening it:

cast_dim2flat(x2)
#> $`['height', 'class1', 'group1']`
#>  [1] 169.1308 168.6513 169.0970 169.9275 168.6425 170.5370 171.2077 169.4859
#>  [9] 169.3879 168.7557
#> 
#> $`['weight', 'class1', 'group1']`
#>  [1] 79.01396 81.36939 80.69187 80.11392 79.34477 79.97995 81.09971 78.10813
#>  [9] 78.22781 79.92416
#> 
#> $`['sex', 'class1', 'group1']`
#>  [1] "F" "M" "F" "F" "F" "F" "F" "M" NA  NA 
#> 
#> $`['height', 'class2', 'group1']`
#> ~"this is padding!"
#> 
#> $`['weight', 'class2', 'group1']`
#> ~"this is padding!"
#> 
#> $`['sex', 'class2', 'group1']`
#> ~"this is padding!"
#> 
#> $`['height', 'class1', 'group2']`
#>  [1] 170.2828 172.2632 168.7990 171.7871 170.1060 168.7741 170.0522 169.7033
#>  [9] 169.5451 168.6665
#> 
#> $`['weight', 'class1', 'group2']`
#>  [1] 80.93280 79.99714 81.97345 79.50892 77.53432 79.38361 79.35040 79.92561
#>  [9] 82.33654 79.77424
#> 
#> $`['sex', 'class1', 'group2']`
#>  [1] "F" "M" "F" "F" "M" "M" NA  "F" "M" "M"
#> 
#> $`['height', 'class2', 'group2']`
#>  [1] 169.0460 168.9878 170.7479 171.8110 170.4234 169.3689 168.7653 169.8761
#>  [9] 170.3767 170.4783
#> 
#> $`['weight', 'class2', 'group2']`
#>  [1] 80.28330 80.10681 81.85980 79.85685 82.26221 80.38828 79.66371 80.04306
#>  [9] 81.24818 79.59075
#> 
#> $`['sex', 'class2', 'group2']`
#>  [1] "F" "F" "M" "F" "F" NA  "F" "F" "F" "M"

 

2.5 Example 4: Comparing in2out with padding

In this example, the same nested list as from the previous example is used, to demonstrate the difference between in2out = TRUE (which is the default), and in2out = FALSE.

Consider first the original list again:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

On the left side the list is casted as dimensional using the default of in2out = TRUE, with proper names assigned.
On the right side the list is casted as dimensional using in2out = FALSE, again with proper names assigned.

x2 <- cast_hier2dim(x)
dimnames(x2) <- hiernames2dimnames(x)
print(x2)
#> , , group1
#> 
#>        class1       class2
#> height numeric,10   NULL  
#> weight numeric,10   NULL  
#> sex    character,10 NULL  
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

x2 <- cast_hier2dim(x, in2out = FALSE)
dimnames(x2) <- hiernames2dimnames(x, in2out = FALSE)
print(x2)
#> , , height
#> 
#>        class1     class2    
#> group1 numeric,10 NULL      
#> group2 numeric,10 numeric,10
#> 
#> , , weight
#> 
#>        class1     class2    
#> group1 numeric,10 NULL      
#> group2 numeric,10 numeric,10
#> 
#> , , sex
#> 
#>        class1       class2      
#> group1 character,10 NULL        
#> group2 character,10 character,10

 

3 Cast Dimensional List to Hierarchical list

‘broadcast’ provides the cast_dim2hier():
cast_dim2hier() takes a dimensional list (i.e. an array of type list), and casts it to a nested list.

Consider the following recursive array as an example:

x <- array(c(as.list(1:11), ~hello, as.list(month.abb)), c(4:2))
dimnames(x) <- list(
  letters[1:4],
  LETTERS[1:3],
  c("group1", "group2")
)
print(x)
#> , , group1
#> 
#>   A B C     
#> a 1 5 9     
#> b 2 6 10    
#> c 3 7 11    
#> d 4 8 ~hello
#> 
#> , , group2
#> 
#>   A     B     C    
#> a "Jan" "May" "Sep"
#> b "Feb" "Jun" "Oct"
#> c "Mar" "Jul" "Nov"
#> d "Apr" "Aug" "Dec"

Like cast_hier2dim() also has the in2out argument, which (again) defaults to TRUE.
Let’s cast the above dimensional list to a nested list, and compare the results when using in2out = TRUE (on the left) versus in2out = FALSE (on the right):

x2 <- cast_dim2hier(
  x, distr.names = TRUE
)
lobstr::tree(x2)
#> <list>
#> ├─group1: <list>
#> │ ├─A: <list>
#> │ │ ├─a: 1
#> │ │ ├─b: 2
#> │ │ ├─c: 3
#> │ │ └─d: 4
#> │ ├─B: <list>
#> │ │ ├─a: 5
#> │ │ ├─b: 6
#> │ │ ├─c: 7
#> │ │ └─d: 8
#> │ └─C: <list>
#> │   ├─a: 9
#> │   ├─b: 10
#> │   ├─c: 11
#> │   └─d: S3<formula> ~hello
#> └─group2: <list>
#>   ├─A: <list>
#>   │ ├─a: "Jan"
#>   │ ├─b: "Feb"
#>   │ ├─c: "Mar"
#>   │ └─d: "Apr"
#>   ├─B: <list>
#>   │ ├─a: "May"
#>   │ ├─b: "Jun"
#>   │ ├─c: "Jul"
#>   │ └─d: "Aug"
#>   └─C: <list>
#>     ├─a: "Sep"
#>     ├─b: "Oct"
#>     ├─c: "Nov"
#>     └─d: "Dec"

x2 <- cast_dim2hier(
  x, in2out = FALSE, distr.names = TRUE
)
lobstr::tree(x2)
#> <list>
#> ├─a: <list>
#> │ ├─A: <list>
#> │ │ ├─group1: 1
#> │ │ └─group2: "Jan"
#> │ ├─B: <list>
#> │ │ ├─group1: 5
#> │ │ └─group2: "May"
#> │ └─C: <list>
#> │   ├─group1: 9
#> │   └─group2: "Sep"
#> ├─b: <list>
#> │ ├─A: <list>
#> │ │ ├─group1: 2
#> │ │ └─group2: "Feb"
#> │ ├─B: <list>
#> │ │ ├─group1: 6
#> │ │ └─group2: "Jun"
#> │ └─C: <list>
#> │   ├─group1: 10
#> │   └─group2: "Oct"
#> ├─c: <list>
#> │ ├─A: <list>
#> │ │ ├─group1: 3
#> │ │ └─group2: "Mar"
#> │ ├─B: <list>
#> │ │ ├─group1: 7
#> │ │ └─group2: "Jul"
#> │ └─C: <list>
#> │   ├─group1: 11
#> │   └─group2: "Nov"
#> └─d: <list>
#>   ├─A: <list>
#>   │ ├─group1: 4
#>   │ └─group2: "Apr"
#>   ├─B: <list>
#>   │ ├─group1: 8
#>   │ └─group2: "Aug"
#>   └─C: <list>
#>     ├─group1: S3<formula> ~hello
#>     └─group2: "Dec"

The added distr.names = TRUE argument will distribute the dimnames in a logical way over the nested elements.

 

4 Simple Data Wrangling Example: Turning list inside out

The cast functions can be used to turn a list inside out.

Let’s start with the following list:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(5, 170) |> as.integer(),
      weight = rnorm(5, 80)  |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    ),
    class2 = list(
      height = rnorm(5, 170)  |> as.integer(),
      weight = rnorm(5, 80) |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(5, 170) |> as.integer(),
      weight = rnorm(5, 80) |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    ),
    class2 = list(
      height = rnorm(5, 170) |> as.integer(),
      weight = rnorm(5, 80) |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    )
  )
)

Turning this list inside out means manipulating this list such that height, weight and sex become the surface-level elements and the groups become the deepest levels.

This can be done fast & easy with ‘broadcast’, by casting the nested list to dimensional with in2out = TRUE, and then casting the dimensional list back to nested using in2out = FALSE.

 

First, cast nested list to dimensional list:

x2 <- cast_hier2dim(x, direction.names = 1)
print(x2)
#> , , group1
#> 
#>        class1      class2     
#> height integer,5   integer,5  
#> weight integer,5   integer,5  
#> sex    character,5 character,5
#> 
#> , , group2
#> 
#>        class1      class2     
#> height integer,5   integer,5  
#> weight integer,5   integer,5  
#> sex    character,5 character,5

The default value for in2out is TRUE, so we don’t have to specify it here.
The direction.names argument (available from broadcast version 0.1.6) in cast_hier2dim() will construct dimensional names for you, so you don’t have to set them manually using hiernames2dimnames().

Second, cast the newly created dimensional list back to a nested list, but this time use in2out = FALSE:

x3 <- cast_dim2hier(x2, in2out = FALSE, distr.names = TRUE)
lobstr::tree(x3)
#> <list>
#> ├─height: <list>
#> │ ├─class1: <list>
#> │ │ ├─group1<int [5]>: 168, 168, 170, 169, 170
#> │ │ └─group2<int [5]>: 171, 171, 170, 170, 169
#> │ └─class2: <list>
#> │   ├─group1<int [5]>: 168, 169, 167, 169, 170
#> │   └─group2<int [5]>: 171, 171, 171, 167, 171
#> ├─weight: <list>
#> │ ├─class1: <list>
#> │ │ ├─group1<int [5]>: 81, 80, 79, 79, 80
#> │ │ └─group2<int [5]>: 80, 80, 78, 79, 79
#> │ └─class2: <list>
#> │   ├─group1<int [5]>: 80, 81, 79, 77, 80
#> │   └─group2<int [5]>: 80, 82, 78, 80, 80
#> └─sex: <list>
#>   ├─class1: <list>
#>   │ ├─group1<chr [5]>: "NA", "F", "NA", "M", "F"
#>   │ └─group2<chr [5]>: "NA", "F", "NA", "NA", "F"
#>   └─class2: <list>
#>     ├─group1<chr [5]>: "F", "NA", "F", "F", "F"
#>     └─group2<chr [5]>: "M", "M", "M", "M", "NA"

We use lobstr::tree() to print the results more compactly;
as shown, the original nested list has now successfully been turned inside-out, with great ease.