Skip to contents

collapse provides an ensemble of functions to perform common data transformations efficiently and user friendly:

  • dapply applies functions to rows or columns of matrices and data frames, preserving the data format.

  • BY is an S3 generic for efficient Split-Apply-Combine computing, similar to dapply.

  • A set of arithmetic operators facilitates row-wise %rr%, %r+%, %r-%, %r*%, %r/% and column-wise %cr%, %c+%, %c-%, %c*%, %c/% replacing and sweeping operations involving a vector and a matrix or data frame / list. Since v1.7, the operators %+=%, %-=%, %*=% and %/=% do column- and element- wise math by reference, and the function setop can also perform sweeping out rows by reference.

  • (set)TRA is a more advanced S3 generic to efficiently perform (groupwise) replacing and sweeping out of statistics, either by creating a copy of the data or by reference. Supported operations are:

    Integer-id String-id Description
    0"na" or "replace_na"replace only missing values
    1"fill" or "replace_fill"replace everything
    2"replace"replace data but preserve missing values
    4"-+"subtract group-statistics but add group-frequency weighted average of group statistics
    6"%"compute percentages
    10"-%%"subtract modulus

    All of collapse's Fast Statistical Functions have a built-in TRA argument for faster access (i.e. you can compute (groupwise) statistics and use them to transform your data with a single function call).

  • fscale/STD is an S3 generic to perform (groupwise and / or weighted) scaling / standardizing of data and is orders of magnitude faster than scale.

  • fwithin/W is an S3 generic to efficiently perform (groupwise and / or weighted) within-transformations / demeaning / centering of data. Similarly fbetween/B computes (groupwise and / or weighted) between-transformations / averages (also a lot faster than ave).

  • fhdwithin/HDW, shorthand for 'higher-dimensional within transform', is an S3 generic to efficiently center data on multiple groups and partial-out linear models (possibly involving many levels of fixed effects and interactions). In other words, fhdwithin/HDW efficiently computes residuals from linear models. Similarly fhdbetween/HDB, shorthand for 'higher-dimensional between transformation', computes the corresponding means or fitted values.

  • flag/L/F, fdiff/D/Dlog and fgrowth/G are S3 generics to compute sequences of lags / leads and suitably lagged and iterated (quasi-, log-) differences and growth rates on time series and panel data. fcumsum flexibly computes (grouped, ordered) cumulative sums. More in Time Series and Panel Series.

  • STD, W, B, HDW, HDB, L, D, Dlog and G are parsimonious wrappers around the f- functions above representing the corresponding transformation 'operators'. They have additional capabilities when applied to data-frames (i.e. variable selection, formula input, auto-renaming and id-variable preservation), and are easier to employ in regression formulas, but are otherwise identical in functionality.

Table of Functions

Function / S3 Generic Methods Description
dapplyNo methods, works with matrices and data framesApply functions to rows or columns
BYdefault, matrix, data.frame, grouped_dfSplit-Apply-Combine computing
%(r/c)(r/+/-/*//)%No methods, works with matrices and data frames / listsRow- and column-arithmetic
(set)TRAdefault, matrix, data.frame, grouped_dfReplace and sweep out statistics (by reference)
fscale/STDdefault, matrix, data.frame, pseries, pdata.frame, grouped_dfScale / standardize data
fwithin/Wdefault, matrix, data.frame, pseries, pdata.frame, grouped_dfDemean / center data
fbetween/Bdefault, matrix, data.frame, pseries, pdata.frame, grouped_dfCompute means / average data
fhdwithin/HDWdefault, matrix, data.frame, pseries, pdata.frameHigh-dimensional centering and lm residuals
fhdbetween/HDBdefault, matrix, data.frame, pseries, pdata.frameHigh-dimensional averages and lm fitted values
flag/L/F, fdiff/D/Dlog, fgrowth/G, fcumsumdefault, matrix, data.frame, pseries, pdata.frame, grouped_df(Sequences of) lags / leads, differences, growth rates and cumulative sums