collapse provides an ensemble of functions to perform common data transformations efficiently and user friendly:

  • dapply applies functions to rows or columns of matrices and data frames, preserving the data format.

  • BY is an S3 generic for Split-Apply-Combine computing and can perform aggregation as well as grouped transformations (for aggregation please also see collap and the Fast Statistical Functions).

  • A set of arithmetic operators facilitates row-wise %rr%, %r+%, %r-%, %r*%, %r/% and column-wise %cr%, %c+%, %c-%, %c*%, %c/% replacing and sweeping operations involving a vector and a matrix or data frame / list.

  • TRA is a more advanced S3 generic to efficiently perform (groupwise) replacing and sweeping out of statistics. Supported operations are:

    Integer-id String-id Description
    1"replace_fill"replace and overwrite missing values
    2"replace"replace but preserve missing values
    3"-"subtract
    4"-+"subtract group-statistics but add group-frequency weighted average of group statistics
    5"/"divide
    6"%"compute percentages
    7"+"add
    8"*"multiply
    9"%%"modulus
    10"-%%"subtract modulus

    All of collapse's Fast Statistical Functions have a built-in TRA argument for faster access (i.e. you can compute (groupwise) statistics and use them to transform your data with a single function call).

  • fscale/STD is an S3 generic to perform (groupwise and / or weighted) scaling / standardizing of data and is orders of magnitude faster than scale.

  • fwithin/W is an S3 generic to efficiently perform (groupwise and / or weighted) within-transformations / demeaning / centering of data. Similarly fbetween/B computes (groupwise and / or weighted) between-transformations / averages (also a lot faster than ave).

  • fHDwithin/HDW, shorthand for 'higher-dimensional within transform', is an S3 generic to efficiently center data on multiple groups and partial-out linear models (possibly involving many levels of fixed effects). In other words, fHDwithin/HDW efficiently computes residuals from (potentially complex) linear models. Similarly fHDbetween/HDB, shorthand for 'higher-dimensional between transformation', computes the corresponding means or fitted values.

  • flag/L/F, fdiff/D/Dlog and fgrowth/G are S3 generics to compute sequences of lags / leads and suitably lagged and iterated (quasi-, log-) differences and growth rates on time series and panel data. More in Time Series and Panel Series.

  • STD, W, B, HDW, HDB, L, D, Dlog and G are parsimonious wrappers around the f- functions above representing the corresponding transformation 'operators'. They have additional capabilities when applied to data-frames (i.e. variable selection, formula input, auto-renaming and id-variable preservation), and are easier to employ in regression formulas, but are otherwise identical in functionality.

Table of Functions

Function / S3 Generic Methods Description
dapplyNo methods, works with matrices and data framesApply functions to rows or columns
BYdefault, matrix, data.frame, grouped_dfSplit-Apply-Combine computing
%(r/c)(r/+/-/*//)%No methods, works with matrices and data frames / listsRow- and column-arithmetic
TRAdefault, matrix, data.frame, grouped_dfReplace and sweep out statistics
fscale/STDdefault, matrix, data.frame, pseries, pdata.frame, grouped_dfScale / standardize data
fwithin/Wdefault, matrix, data.frame, pseries, pdata.frame, grouped_dfDemean / center data
fbetween/Bdefault, matrix, data.frame, pseries, pdata.frame, grouped_dfCompute means / average data
fHDwithin/HDWdefault, matrix, data.frame, pseries, pdata.frameHigh-dimensional centering and lm residuals
fHDbetween/HDBdefault, matrix, data.frame, pseries, pdata.frameHigh-dimensional averages and lm fitted values
flag/L/Fdefault, matrix, data.frame, pseries, pdata.frame, grouped_df(Sequences of) lags / leads
fdiff/D/Dlogdefault, matrix, data.frame, pseries, pdata.frame, grouped_df(Sequences of lagged/leaded and iterated quasi- log-) differences
fgrowth/Gdefault, matrix, data.frame, pseries, pdata.frame, grouped_df(Sequences of lagged/leaded and iterated) growth rates

See also