What even is longitudinal data?

What even is longitudinal data?

What even is longitudinal data?

What even is longitudinal data?

All of Australia

...And New Zealand

And the rest?

Problems:

Answers: Transparency?

Answers: Transparency + a model?

But we forget about the individuals

Introducing brolgar: brolgar.njtierney.com

Longitudinal data as a time series

heights <- as_tsibble(heights,
                      index = year,
                      key = country,
                      regular = FALSE)

Longitudinal data as a time series

Portion out your spaghetti! 🍝 🍝 🍝 🍝

Look at one set of subsamples 🍝

Look at many subsamples 🍝 🍝 🍝 🍝 🍝 🍝

Distraction threshold ⏲ 🐰 🕳

Avoiding the rabbit hole

facet_sample(): See more individuals 👀

ggplot(heights, aes(x = year, 
                    y = height_cm, 
                    group = country)) + 
  geom_line()

facet_sample(): See more individuals 👀

facet_sample(): See more individuals 👀

facet_sample(): See more individuals 👀

ggplot(heights, aes(x = year,
                    y = height_cm,
                    group = country)) + 
  geom_line() + 
  facet_sample()

facet_sample(): See more individuals 👀

Remove distraction. Ask relevant questions

gg_heights +  
  facet_sample(
    n_per_facet = 3,
    n_facets = 9
  )

How to see all individuals?

gg_heights +  
  facet_strata()

facet_strata(): See all individuals

🤔 ... can we re-order these facets?

We can re-order these facets?! 😄

See all individuals along some variable

gg_heights + 
  facet_strata(
    along = -year
    )

See all individuals along some variable

Magic facets: Focus on relevant questions instead of minutiae:

gg_heights +  
  facet_sample(
    n_per_facet = 10,
    n_facets = 12
    )

gg_heights +  
  facet_strata(
    n_strata = 10,
    along = -year
    )

facet_strata() & facet_sample()

Problem #1: How do I look at some of the data?

Problem #1: How do I look at some of the data?

Problem #2: How do I find interesting observations?

Identify features: one observation per key

Identify features: one observation per key

Identify features: one observation per key

Identify important features and decide how to filter

Identify important features and decide how to filter

Join this feature back to the data

Join this feature back to the data

🎉 Countries with smallest and largest max height

Identify features: one observation per key

## # A tsibble: 1,490 x 3 [!]
## # Key:       country [144]
##   country      year height_cm
##   <chr>       <dbl>     <dbl>
## 1 Afghanistan  1870      168.
## 2 Afghanistan  1880      166.
## 3 Afghanistan  1930      167.
## 4 Afghanistan  1990      167.
## 5 Afghanistan  2000      161.
## 6 Albania      1880      170.
## # … with 1,484 more rows

Identify features: one observation per key

## # A tibble: 144 x 2
##   country       max
##   <chr>       <dbl>
## 1 Afghanistan  168.
## 2 Albania      170.
## 3 Algeria      171.
## 4 Angola       169.
## 5 Argentina    174.
## 6 Armenia      172.
## # … with 138 more rows

Identify important features and decide how to filter

heights_five %>% 
  filter(max == max(max) | max == min(max))

## # A tibble: 2 x 2
##   country            max
##   <chr>            <dbl>
## 1 Denmark           183.
## 2 Papua New Guinea  161.

Join summaries back to data

heights_five %>% 
  filter(max == max(max) | max == min(max)) %>% 
  left_join(heights, by = "country")

## # A tibble: 21 x 4
##   country   max  year height_cm
##   <chr>   <dbl> <dbl>     <dbl>
## 1 Denmark  183.  1820      167.
## 2 Denmark  183.  1830      165.
## 3 Denmark  183.  1850      167.
## 4 Denmark  183.  1860      168.
## 5 Denmark  183.  1870      168.
## 6 Denmark  183.  1880      170.
## # … with 15 more rows

Identify features: one per key

heights %>%
  features(height_cm, feat_five_num)

## # A tibble: 144 x 6
##   country       min   q25   med   q75   max
##   <chr>       <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 Afghanistan  161.  164.  167.  168.  168.
## 2 Albania      168.  168.  170.  170.  170.
## 3 Algeria      166.  168.  169   170.  171.
## 4 Angola       159.  160.  167.  168.  169.
## 5 Argentina    167.  168.  168.  170.  174.
## 6 Armenia      164.  166.  169.  172.  172.
## # … with 138 more rows

What is the range of the data? feat_ranges

heights %>%
  features(height_cm, feat_ranges)

## # A tibble: 144 x 5
##   country       min   max range_diff   iqr
##   <chr>       <dbl> <dbl>      <dbl> <dbl>
## 1 Afghanistan  161.  168.       7     3.27
## 2 Albania      168.  170.       2.20  1.53
## 3 Algeria      166.  171.       5.06  2.15
## 4 Angola       159.  169.      10.5   7.87
## 5 Argentina    167.  174.       7     2.21
## 6 Armenia      164.  172.       8.82  5.30
## # … with 138 more rows

Does it only increase or decrease? feat_monotonic

heights %>%
  features(height_cm, feat_monotonic)

## # A tibble: 144 x 5
##   country     increase decrease unvary monotonic
##   <chr>       <lgl>    <lgl>    <lgl>  <lgl>    
## 1 Afghanistan FALSE    FALSE    FALSE  FALSE    
## 2 Albania     FALSE    TRUE     FALSE  TRUE     
## 3 Algeria     FALSE    FALSE    FALSE  FALSE    
## 4 Angola      FALSE    FALSE    FALSE  FALSE    
## 5 Argentina   FALSE    FALSE    FALSE  FALSE    
## 6 Armenia     FALSE    FALSE    FALSE  FALSE    
## # … with 138 more rows

What is the spread of my data? feat_spread

heights %>%
  features(height_cm, feat_spread)

## # A tibble: 144 x 5
##   country        var    sd   mad   iqr
##   <chr>        <dbl> <dbl> <dbl> <dbl>
## 1 Afghanistan  7.20  2.68  1.65   3.27
## 2 Albania      0.950 0.975 0.667  1.53
## 3 Algeria      3.30  1.82  0.741  2.15
## 4 Angola      16.9   4.12  3.11   7.87
## 5 Argentina    2.89  1.70  1.36   2.21
## 6 Armenia     10.6   3.26  3.60   5.30
## # … with 138 more rows

features: MANY more features in feasts

Problem #1: How do I look at some of the data?

Problem #2: How do I find interesting observations?

Problem #1: How do I look at some of the data?

Problem #2: How do I find interesting observations?

Problem #3: How do I understand a model?

Problem #3: How do I understand a model?

heights_fit <- lmer(height_cm ~ year + (1|country), heights)
heights_aug <- heights %>%
  add_predictions(heights_fit, var = "pred") %>%
  add_residuals(heights_fit, var = "res")

Problem #3: How do I understand a model?

## # A tsibble: 1,490 x 5 [!]
## # Key:       country [144]
##   country      year height_cm  pred    res
##   <chr>       <dbl>     <dbl> <dbl>  <dbl>
## 1 Afghanistan  1870      168.  164.  4.59 
## 2 Afghanistan  1880      166.  164.  1.52 
## 3 Afghanistan  1930      167.  166.  0.823
## 4 Afghanistan  1990      167.  168. -1.04 
## 5 Afghanistan  2000      161.  169. -7.10 
## 6 Albania      1880      170.  168.  2.39 
## # … with 1,484 more rows

Problem #3: How do I understand a model?

Look at many subsamples? facet_sample()

gg_heights_fit + facet_sample()

Look at all subsamples? facet_strata()

gg_heights_fit + facet_strata()

Look at all subsamples along residuals?

gg_heights_fit + facet_strata(along = -res)

Look at the predictions with the data?

set.seed(2020-01-21)
heights_sample <- 
heights_aug %>%
  sample_n_keys(size = 9) %>%
  ggplot(aes(x = year,
             y = pred,
             group = country)) + 
  geom_line() +
  facet_wrap(~country)
heights_sample

Look at the predictions with the data?

Look at the predictions with the data?

heights_sample + geom_point(aes(y = height_cm))

Take homes

Problem #1: How do I look at some of the data?

Take homes

Problem #2: How do I find interesting observations?

Take homes

Problem #3: How do I understand a model?

Future Directions

Thanks

Resources

Colophon

Learning more

What about interactive graphics?

What if we grabbed a sample of those who have the best, middle, and worst residuals?

summary(heights_aug$res)

##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
## -8.1707 -1.6202 -0.1558  0.0000  1.3545 12.1729

use keys_near()

keys_near(heights_aug, 
          var = res)

## # A tibble: 6 x 5
##   country       res stat  stat_value stat_diff
##   <chr>       <dbl> <fct>      <dbl>     <dbl>
## 1 Ireland    -8.17  min       -8.17   0       
## 2 Azerbaijan -1.62  q_25      -1.62   0.000269
## 3 Laos       -0.157 med       -0.156  0.00125 
## 4 Mongolia   -0.155 med       -0.156  0.00125 
## 5 Egypt       1.35  q_75       1.35   0.000302
## 6 Poland     12.2   max       12.2    0

Join data back and explore