List Casting Explained

1 Introduction

Hierarchical data is surprisingly common, and are commonly represented in by nested lists.

Broadcasted operations can be performed over dimensions, but not through nesting or hierarchies.
Therefore, it is useful to be able to cast nested lists into dimensional lists.
The ‘broadcast’ package provides the cast_hier2dim() to cast a nested list into a dimensional list (AKA a recursive array).

Casting between nested and dimensional lists is not only useful for broadcasting, however.
Casting nested lists to dimensional lists has its own merits, as dimensional lists have some advantages over nested lists beside the broadcasting, such as the following:

• Performing sub-set operations on multiple recursive subsets (using the [[ and [[<- operators) requires a (potentially slow) loop, whereas multi-dimensional subsets (using operator forms like [..., ...] and [..., ...]<-) are vectorized and generally much faster.
• Re-organizing dimensions of a recursive array is generally much easier, faster, and more straight-forward than re-organizing hierarchies of a nested list.

This Vignette gives an overview of the functions ‘broadcast’ provides to cast between nested and dimensional lists.

 

2 Cast Hierarchical List to Dimensional List

2.1 Introduction

The cast_hier2dim() function casts a nested list into a dimensional list.
This section gently introduces the properties of this function through a series of examples, where each subsequent example builds on the previous one.
Familiarity with nested lists and dimensional lists (i.e. arrays of type list) is essential to follow these examples.

 

2.2 Example 1: Basics

2.2.1 Casting

For a first example, consider the following list:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

Before actually casting x into a dimensional list, one may want to know what the dimensions will become when casted as a dimensional list;
The hier2dim() function shows you that:

hier2dim(x)
#>       
#> 3 2 2

It returns the dimensions c(3, 2, 2).

Let’s now cast x as a dimensional list:

x2 <- cast_hier2dim(x) # actually cast nested list into dimensional list
print(x2)
#> , , 1
#> 
#>      [,1]         [,2]        
#> [1,] numeric,10   numeric,10  
#> [2,] numeric,10   numeric,10  
#> [3,] character,10 character,10
#> 
#> , , 2
#> 
#>      [,1]         [,2]        
#> [1,] numeric,10   numeric,10  
#> [2,] numeric,10   numeric,10  
#> [3,] character,10 character,10

Using the default arguments, element x[[i]][[j]][[k]] corresponds to element x2[k, j, i] (for all i, j, and k).
This can be changed, as will be shown in a later example.

 

2.2.2 Naming

As shown in the results above, cast_hier2dim() will not preserve names by default.
This is because there is no guaranteed unique way to distribute nested names over dimensional names.
It is trivially easy to set the dimnames of x2 yourself, using hiernames2dimnames():

dimnames(x2) <- hiernames2dimnames(x, direction = 1)
print(x2)
#> , , group1
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

Alternatively, you can tell cast_hier2dim() to attempt to choose logical dimensional names, using the direction.names argument (this argument corresponds to the direction argument of hiernames2dimnames()):

x2 <- cast_hier2dim(x, direction.names = 1)
print(x2)
#> , , group1
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

 

2.2.3 Printing

will display a dimensional list more compactly than a nested list.
Depending on the situation this may be either be desirable or undesirable.

One can print x2 less compactly without much effort by flattening it, using the cast_dim2flat() function.
We only need to see a portion of the list in detail, so let’s look at class1 from group 1 in the flattened form:

cast_dim2flat(x2[, 1, "group1", drop = FALSE])
#> $`['height', 'class1', 'group1']`
#>  [1] 168.7803 171.3005 171.4129 169.9441 170.7197 171.5639 168.7416 168.5182
#>  [9] 170.3498 170.9825
#> 
#> $`['weight', 'class1', 'group1']`
#>  [1] 80.68647 80.14773 79.28123 80.73879 80.15146 79.95663 80.79686 82.17036
#>  [9] 79.75453 79.99322
#> 
#> $`['sex', 'class1', 'group1']`
#>  [1] "F" "M" "M" "M" "M" "M" "M" NA  "F" "F"

 

2.2.4 Application

Dimensional lists can be easier to work with than hierarchical lists.
Consider, for example, printing the height of the first class of every group in a list - let’s compare how to do this in a nested list vs a dimensional list.

With a nested list, doing this takes a slow, messy for-loop:

for(i in seq_along(x)) {
  print(names(x)[i])
  x[[i]][[1]][["height"]] |> print() # slow for-loop, messy code
}
#> [1] "group1"
#>  [1] 168.7803 171.3005 171.4129 169.9441 170.7197 171.5639 168.7416 168.5182
#>  [9] 170.3498 170.9825
#> [1] "group2"
#>  [1] 169.4836 169.8066 168.9157 169.4493 172.0195 168.9464 169.7807 171.5298
#>  [9] 171.1720 170.2164

With a dimensional list, the very same thing can be done with sleek, vectorized code; no messy loop needed:

x2["height", 1L, ] |> print()
#> $group1
#>  [1] 168.7803 171.3005 171.4129 169.9441 170.7197 171.5639 168.7416 168.5182
#>  [9] 170.3498 170.9825
#> 
#> $group2
#>  [1] 169.4836 169.8066 168.9157 169.4493 172.0195 168.9464 169.7807 171.5298
#>  [9] 171.1720 170.2164

x2["height", 1L, , drop = FALSE] |> cast_dim2flat() # same but more informative
#> $`['height', 'class1', 'group1']`
#>  [1] 168.7803 171.3005 171.4129 169.9441 170.7197 171.5639 168.7416 168.5182
#>  [9] 170.3498 170.9825
#> 
#> $`['height', 'class1', 'group2']`
#>  [1] 169.4836 169.8066 168.9157 169.4493 172.0195 168.9464 169.7807 171.5298
#>  [9] 171.1720 170.2164

It is also easier to re-arrange dimensions - for example using aperm() - than it is to re-arrange hierarchies.

 

2.3 Example 2: Cast from outside to inside

In Example 1, the default arguments were used for cast_hier2dim().
One of these arguments is in2out, which defaults to TRUE.

Consider a nested list x with a depth of 3, and a dimensional list X2 with 3 dimensions, where the relationship between x and x2 can be expressed as x2 <- cast_hier2dim(x, ...).
Given this, the following can be stated about in2out:

• If in2out = TRUE, which is the default and used in Example 1, element x[[i]][[j]][[k]] corresponds to element x2[k, j, i] (for all i, j, and k).
  
• If in2out = FALSE, element x[[i]][[j]][[k]] corresponds to element x2[i, j, k] (for all i, j, and k).

The default of in2out = TRUE was chosen, because elements in subsequent rows are close to each other, while elements in subsequent layers (third dimension) are generally not close to each other, and the default of in2out = TRUE attempts to retain that behaviour.

For this example, the same list will be used as in Example 1:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

Let’s once again cast this list to a dimensional list, but this time use in2out = FALSE:

hier2dim(x, in2out = FALSE) # check once again the dimensions
#>       
#> 2 2 3
x2 <- cast_hier2dim(x, in2out = FALSE, direction.names = 1)
print(x2)
#> , , height
#> 
#>        class1     class2    
#> group1 numeric,10 numeric,10
#> group2 numeric,10 numeric,10
#> 
#> , , weight
#> 
#>        class1     class2    
#> group1 numeric,10 numeric,10
#> group2 numeric,10 numeric,10
#> 
#> , , sex
#> 
#>        class1       class2      
#> group1 character,10 character,10
#> group2 character,10 character,10

x2 is the casted list. Since in2out = FALSE, element x[[i]][[j]][[k]] corresponds to element x2[i, j, k] (for all i, j, and k).
The argument direction.names = 1 was specified, which intelligently tries to deduce good dimnames for the result. So we don’t have to set the dimension names via hiernames2dimnames().

One can print x2 less compactly without much effort by flattening it, again using the cast_dim2flat() function.
We only need to see a portion of the list in detail, so let’s look at class1 from group 1 in the flattened form:

cast_dim2flat(x2["group1", 1, , drop = FALSE])
#> $`['group1', 'class1', 'height']`
#>  [1] 170.1720 171.2672 170.3443 168.7900 170.2764 169.3781 170.0621 171.2967
#>  [9] 168.3075 171.3656
#> 
#> $`['group1', 'class1', 'weight']`
#>  [1] 78.76268 80.21827 80.37192 79.88133 80.64715 79.93649 78.53107 79.77299
#>  [9] 79.19927 80.70970
#> 
#> $`['group1', 'class1', 'sex']`
#>  [1] NA  NA  "F" "F" "F" "M" "M" NA  "M" "M"

 

2.4 Example 3: Padding

For Example 3, we take the same list as before, but remove x$group1$class2:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

Let’s first check what dimensions it will get when casted using hier2dim():

hier2dim(x)
#>         padding         
#>       3       2       2

The dimensions are the same as in Example 1: c(3, 2, 2).
But notice the names of the output are different: the second element has the name “padding”; this indicates that some columns won’t have enough elements to completely fill the column, and so additional elements will be added as padding.

So let’s cast this list as dimensional:

x2 <- cast_hier2dim(x, direction.names = 1)
print(x2)
#> , , group1
#> 
#>        class1       class2
#> height numeric,10   NULL  
#> weight numeric,10   NULL  
#> sex    character,10 NULL  
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

Subset x2[, 2, 1] is filled with NULL; this is the place where x$group1$class2 was in Example 1, but since it’s not there, we need to fill something.

Sometimes, a different value than NULL is desired for padding.
So let’s replace the padding value with something really obvious, using the padding argument:

x2 <- cast_hier2dim(x, padding = list(~ "this is padding!"), direction.names = 1)
print(x2)
#> , , group1
#> 
#>        class1       class2             
#> height numeric,10   ~"this is padding!"
#> weight numeric,10   ~"this is padding!"
#> sex    character,10 ~"this is padding!"
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

Once again, one can print or present x2 less compactly by flattening it:

cast_dim2flat(x2)
#> $`['height', 'class1', 'group1']`
#>  [1] 170.8444 169.0533 171.1156 170.4087 170.2124 169.6596 168.4781 169.2053
#>  [9] 170.3494 171.0768
#> 
#> $`['weight', 'class1', 'group1']`
#>  [1] 79.54362 80.09165 80.62859 79.84927 77.91781 79.85341 79.64819 80.92041
#>  [9] 79.91065 80.48743
#> 
#> $`['sex', 'class1', 'group1']`
#>  [1] "F" "F" NA  NA  NA  "F" NA  "M" "M" "F"
#> 
#> $`['height', 'class2', 'group1']`
#> ~"this is padding!"
#> 
#> $`['weight', 'class2', 'group1']`
#> ~"this is padding!"
#> 
#> $`['sex', 'class2', 'group1']`
#> ~"this is padding!"
#> 
#> $`['height', 'class1', 'group2']`
#>  [1] 170.5719 169.9235 169.9957 170.8888 170.3310 170.3580 169.3028 171.9603
#>  [9] 170.4032 170.7553
#> 
#> $`['weight', 'class1', 'group2']`
#>  [1] 79.67561 82.09965 79.09383 79.27880 79.39016 79.41172 77.78418 78.31881
#>  [9] 79.89055 79.90574
#> 
#> $`['sex', 'class1', 'group2']`
#>  [1] NA  "M" "F" NA  "F" "M" "M" "M" "M" "M"
#> 
#> $`['height', 'class2', 'group2']`
#>  [1] 167.8610 169.0660 169.9795 170.6718 171.9951 168.6015 170.1451 170.2100
#>  [9] 168.0976 170.3574
#> 
#> $`['weight', 'class2', 'group2']`
#>  [1] 79.25872 79.23511 80.10052 80.16062 81.96665 80.18575 81.33429 79.48277
#>  [9] 80.89473 79.39376
#> 
#> $`['sex', 'class2', 'group2']`
#>  [1] "M" "M" NA  "M" "F" "M" NA  "M" "M" "M"

 

2.5 Example 4: Comparing in2out with padding

In this example, the same nested list as from the previous example is used, to demonstrate the difference between in2out = TRUE (which is the default), and in2out = FALSE.

Consider first the original list again:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    ),
    class2 = list(
      height = rnorm(10, 170),
      weight = rnorm(10, 80),
      sex = sample(c("M", "F", NA), 10, TRUE)
    )
  )
)

On the left side the list is casted as dimensional using the default of in2out = TRUE, with proper names assigned.
On the right side the list is casted as dimensional using in2out = FALSE, again with proper names assigned.
For the sake of this example, we will set the dimnames manually via hiernames2dimnames().

x2 <- cast_hier2dim(x)
dimnames(x2) <- hiernames2dimnames(x)
print(x2)
#> , , group1
#> 
#>        class1       class2
#> height numeric,10   NULL  
#> weight numeric,10   NULL  
#> sex    character,10 NULL  
#> 
#> , , group2
#> 
#>        class1       class2      
#> height numeric,10   numeric,10  
#> weight numeric,10   numeric,10  
#> sex    character,10 character,10

x2 <- cast_hier2dim(x, in2out = FALSE)
dimnames(x2) <- hiernames2dimnames(x, in2out = FALSE)
print(x2)
#> , , height
#> 
#>        class1     class2    
#> group1 numeric,10 NULL      
#> group2 numeric,10 numeric,10
#> 
#> , , weight
#> 
#>        class1     class2    
#> group1 numeric,10 NULL      
#> group2 numeric,10 numeric,10
#> 
#> , , sex
#> 
#>        class1       class2      
#> group1 character,10 NULL        
#> group2 character,10 character,10

 

3 Cast Dimensional List to Hierarchical list

‘broadcast’ provides the cast_dim2hier():
cast_dim2hier() takes a dimensional list (i.e. an array of type list), and casts it to a nested list.

Consider the following recursive array as an example:

x <- array(c(as.list(1:11), ~hello, as.list(month.abb)), c(4:2))
dimnames(x) <- list(
  letters[1:4],
  LETTERS[1:3],
  c("group1", "group2")
)
print(x)
#> , , group1
#> 
#>   A B C     
#> a 1 5 9     
#> b 2 6 10    
#> c 3 7 11    
#> d 4 8 ~hello
#> 
#> , , group2
#> 
#>   A     B     C    
#> a "Jan" "May" "Sep"
#> b "Feb" "Jun" "Oct"
#> c "Mar" "Jul" "Nov"
#> d "Apr" "Aug" "Dec"

Like cast_hier2dim() also has the in2out argument, which (again) defaults to TRUE.
Let’s cast the above dimensional list to a nested list, and compare the results when using in2out = TRUE (on the left) versus in2out = FALSE (on the right):

x2 <- cast_dim2hier(
  x, distr.names = TRUE
)
lobstr::tree(x2)
#> <list>
#> ├─group1: <list>
#> │ ├─A: <list>
#> │ │ ├─a: 1
#> │ │ ├─b: 2
#> │ │ ├─c: 3
#> │ │ └─d: 4
#> │ ├─B: <list>
#> │ │ ├─a: 5
#> │ │ ├─b: 6
#> │ │ ├─c: 7
#> │ │ └─d: 8
#> │ └─C: <list>
#> │   ├─a: 9
#> │   ├─b: 10
#> │   ├─c: 11
#> │   └─d: S3<formula> ~hello
#> └─group2: <list>
#>   ├─A: <list>
#>   │ ├─a: "Jan"
#>   │ ├─b: "Feb"
#>   │ ├─c: "Mar"
#>   │ └─d: "Apr"
#>   ├─B: <list>
#>   │ ├─a: "May"
#>   │ ├─b: "Jun"
#>   │ ├─c: "Jul"
#>   │ └─d: "Aug"
#>   └─C: <list>
#>     ├─a: "Sep"
#>     ├─b: "Oct"
#>     ├─c: "Nov"
#>     └─d: "Dec"

x2 <- cast_dim2hier(
  x, in2out = FALSE, distr.names = TRUE
)
lobstr::tree(x2)
#> <list>
#> ├─a: <list>
#> │ ├─A: <list>
#> │ │ ├─group1: 1
#> │ │ └─group2: "Jan"
#> │ ├─B: <list>
#> │ │ ├─group1: 5
#> │ │ └─group2: "May"
#> │ └─C: <list>
#> │   ├─group1: 9
#> │   └─group2: "Sep"
#> ├─b: <list>
#> │ ├─A: <list>
#> │ │ ├─group1: 2
#> │ │ └─group2: "Feb"
#> │ ├─B: <list>
#> │ │ ├─group1: 6
#> │ │ └─group2: "Jun"
#> │ └─C: <list>
#> │   ├─group1: 10
#> │   └─group2: "Oct"
#> ├─c: <list>
#> │ ├─A: <list>
#> │ │ ├─group1: 3
#> │ │ └─group2: "Mar"
#> │ ├─B: <list>
#> │ │ ├─group1: 7
#> │ │ └─group2: "Jul"
#> │ └─C: <list>
#> │   ├─group1: 11
#> │   └─group2: "Nov"
#> └─d: <list>
#>   ├─A: <list>
#>   │ ├─group1: 4
#>   │ └─group2: "Apr"
#>   ├─B: <list>
#>   │ ├─group1: 8
#>   │ └─group2: "Aug"
#>   └─C: <list>
#>     ├─group1: S3<formula> ~hello
#>     └─group2: "Dec"

The added distr.names = TRUE argument will distribute the dimnames in a logical way over the nested elements.

 

4 Simple Data Wrangling Example: Turning list inside out

The cast functions can be used to turn a list inside out.

Let’s start with the following list:

x <- list(
  group1 = list(
    class1 = list(
      height = rnorm(5, 170) |> as.integer(),
      weight = rnorm(5, 80)  |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    ),
    class2 = list(
      height = rnorm(5, 170)  |> as.integer(),
      weight = rnorm(5, 80) |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    )
  ),
  group2 = list(
    class1 = list(
      height = rnorm(5, 170) |> as.integer(),
      weight = rnorm(5, 80) |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    ),
    class2 = list(
      height = rnorm(5, 170) |> as.integer(),
      weight = rnorm(5, 80) |> as.integer(),
      sex = sample(c("M", "F", NA), 5, TRUE)
    )
  )
)

Turning this list inside out means manipulating this list such that height, weight and sex become the surface-level elements and the groups become the deepest levels.

This can be done fast & easy with ‘broadcast’, by casting the nested list to dimensional with in2out = TRUE, and then casting the dimensional list back to nested using in2out = FALSE.

 

First, cast nested list to dimensional list:

x2 <- cast_hier2dim(x, direction.names = 1)
print(x2)
#> , , group1
#> 
#>        class1      class2     
#> height integer,5   integer,5  
#> weight integer,5   integer,5  
#> sex    character,5 character,5
#> 
#> , , group2
#> 
#>        class1      class2     
#> height integer,5   integer,5  
#> weight integer,5   integer,5  
#> sex    character,5 character,5

The default value for in2out is TRUE, so we don’t have to specify it here.
The direction.names argument in cast_hier2dim() will construct dimensional names for you, so you don’t have to set them manually using hiernames2dimnames().

Second, cast the newly created dimensional list back to a nested list, but this time use in2out = FALSE:

x3 <- cast_dim2hier(x2, in2out = FALSE, distr.names = TRUE)
lobstr::tree(x3)
#> <list>
#> ├─height: <list>
#> │ ├─class1: <list>
#> │ │ ├─group1<int [5]>: 169, 171, 169, 172, 168
#> │ │ └─group2<int [5]>: 169, 169, 169, 169, 170
#> │ └─class2: <list>
#> │   ├─group1<int [5]>: 169, 171, 170, 170, 170
#> │   └─group2<int [5]>: 169, 169, 170, 169, 170
#> ├─weight: <list>
#> │ ├─class1: <list>
#> │ │ ├─group1<int [5]>: 80, 80, 79, 80, 79
#> │ │ └─group2<int [5]>: 79, 79, 80, 78, 79
#> │ └─class2: <list>
#> │   ├─group1<int [5]>: 79, 80, 80, 79, 80
#> │   └─group2<int [5]>: 79, 80, 78, 81, 79
#> └─sex: <list>
#>   ├─class1: <list>
#>   │ ├─group1<chr [5]>: "M", "F", "NA", "M", "NA"
#>   │ └─group2<chr [5]>: "NA", "F", "NA", "F", "F"
#>   └─class2: <list>
#>     ├─group1<chr [5]>: "F", "M", "NA", "M", "M"
#>     └─group2<chr [5]>: "F", "F", "F", "NA", "NA"

We use lobstr::tree() to print the results more compactly;
as shown, the original nested list has now successfully been turned inside-out, with great ease.