qF, shorthand for 'quick-factor' implements very fast factor generation from atomic vectors using either radix ordering or index hashing followed by sorting.
qG, shorthand for 'quick-group', generates a kind of factor-light without the levels attribute but instead an attribute providing the number of levels. Optionally the levels / groups can be attached, but without converting them to character (which can have large performance implications). Objects have a class 'qG'.
finteraction generates a factor or 'qG' object by interacting multiple vectors or factors. In that process missing values are always replaced with a level and unused levels/combinations are always dropped.
collapse internally makes optimal use of factors and 'qG' objects when passed as grouping vectors to statistical functions (
t arguments) i.e. typically no further grouping or ordering is performed and objects are used directly by statistical C/C++ code.
qF(x, ordered = FALSE, na.exclude = TRUE, sort = TRUE, drop = FALSE, keep.attr = TRUE, method = "auto") qG(x, ordered = FALSE, na.exclude = TRUE, sort = TRUE, return.groups = FALSE, method = "auto") is_qG(x) as_factor_qG(x, ordered = FALSE, na.exclude = TRUE) finteraction(..., factor = TRUE, ordered = FALSE, sort = factor, method = "auto") itn(...) # Shorthand for finteraction
a atomic vector, factor or quick-group.
logical. Adds a class 'ordered'.
TRUE preserves missing values (i.e. no level is generated for
FALSE attaches an additional class
"na.included" which is used to skip missing value checks performed before sending objects to C/C++. See Details.
TRUE sorts the levels in ascending order (like
FALSE provides the levels in order of first appearance, which can be significantly faster. Note that if a factor is passed as input, only
sort = FALSE takes effect and unused levels will be dropped (as factors usually have sorted levels and checking sortedness can be expensive).
x is a factor,
TRUE efficiently drops unused factor levels beforehand using
x has additional attributes apart from 'levels' and 'class', these are preserved in the conversion to factor.
an integer or character string specifying the method of computation:
|1||"auto"||automatic selection: |
|2||"radix"||use radix ordering to generate factors. Supports |
|3||"hash"||use hashing to generate factors. Since v1.8.3 this is a fast hybrid implementation using |
|4||"rcpp_hash"||the previous "hash" algorithm prior to v1.8.3: uses |
Note that for
method = "hash" is always unsorted and
method = "rcpp_hash" is not available.
TRUE returns the unique elements / groups / levels of
x in an attribute called
qF, they are not converted to character.
TRUE returns an factor,
FALSE returns a 'qG' object.
multiple atomic vectors or factors, or a single list of equal-length vectors or factors. See Details.
Whenever a vector is passed to a Fast Statistical Function such as
fmean(mtcars, mtcars$cyl), is is grouped using
use.g.names = FALSE.
qF is a combination of
factor. Applying it to a vector i.e.
qF(x) gives the same result as
qF(x, ordered = TRUE) generates an ordered factor (same as
factor(x, ordered = TRUE)), and
qF(x, na.exclude = FALSE) generates a level for missing values (same as
factor(x, exclude = NULL)). An important addition is that
qF(x, na.exclude = FALSE) also adds a class 'na.included'. This prevents collapse functions from checking missing values in the factor, and is thus computationally more efficient. Therefore factors used in grouped operations should preferably be generated using
qF(x, na.exclude = FALSE). Setting
sort = FALSE gathers the levels in first-appearance order (unless
method = "radix" and
x is numeric, in which case the levels are always sorted). This often gives a noticeable speed improvement.
There are 3 internal methods of computation: radix ordering, hashing, and Rcpp sugar hashing. Radix ordering is done by combining the functions
groupid. It is generally faster than hashing for large numeric data and pre-sorted data (although there are exceptions). Hashing uses
group, followed by
radixorder on the unique elements if
sort = TRUE. It is generally fastest for character data. Rcpp hashing uses
Rcpp::sugar::match. This is often less efficient than the former on large data, but the sorting properties (relying on
std::sort) may be superior in borderline cases where
radixorder fails to deliver exact lexicographic ordering of factor levels.
Regarding speed: In general
qF is around 5x faster than
as.factor on character data and about 30x faster on numeric data. Automatic method dispatch typically does a good job delivering optimal performance.
qG is in the first place a programmers function. It generates a factor-'light' class 'qG' consisting of only an integer grouping vector and an attribute providing the number of groups. It is slightly faster and more memory efficient than
GRP for grouping atomic vectors, and also convenient as it can be stored in a data frame column, which are the main reasons for its existence.
finteraction is simply a wrapper around
as_factor_GRP(GRP.default(X)), where X is replaced by the arguments in '...' combined in a list (so its not really an interaction function but just a multivariate grouping converted to factor, see
GRP for computational details). In general: All vectors, factors, or lists of vectors / factors passed can be interacted. Interactions always create a level for missing values and always drop unused levels.
qF returns an (ordered) factor.
qG returns an object of class 'qG': an integer grouping vector with an attribute
"N.groups" indicating the number of groups, and, if
return.groups = TRUE, an attribute
"groups" containing the vector of unique groups / elements in
x corresponding to the integer-id.
finteraction can return either.
An efficient alternative for character vectors with multithreading support is provided by
qG(x, sort = FALSE, na.exclude = FALSE, method = "hash") internally calls
group(x) which can also be used directly and also supports multivariate groupings where
x can be a list of vectors.
qG reorder groups / factor levels. An exception was added in v1.7, when calling
qF(f, sort = FALSE) on a factor
f, the levels are recast in first appearance order. These objects can however be converted into one another using
qF/qG or the direct method
as_factor_qG (called inside
qF). It is also possible to add a class 'ordered' (
ordered = TRUE) and to create am extra level / integer for missing values (
na.exclude = FALSE) if factors or 'qG' objects are passed to
cylF <- qF(mtcars$cyl) # Factor from atomic vector cylG <- qG(mtcars$cyl) # Quick-group from atomic vector cylG # See the simple structure of this object #>  2 2 1 2 3 2 3 1 1 2 2 3 3 3 3 3 3 1 1 1 1 3 3 3 3 1 1 1 3 2 3 1 #> attr(,"N.groups") #>  3 #> attr(,"class") #>  "qG" cf <- qF(wlddev$country) # Bigger data cf2 <- qF(wlddev$country, na.exclude = FALSE) # With na.included class dat <- num_vars(wlddev) # cf2 is faster in grouped operations because no missing value check is performed library(microbenchmark) microbenchmark(fmax(dat, cf), fmax(dat, cf2)) #> Unit: microseconds #> expr min lq mean median uq max neval #> fmax(dat, cf) 93.193 95.8580 122.8618 97.8055 100.655 1788.297 100 #> fmax(dat, cf2) 86.756 91.2045 102.7140 92.9470 95.448 426.564 100 finteraction(mtcars$cyl, mtcars$vs) # Interacting two variables (can be factors) #>  6.0 6.0 4.1 6.1 8.0 6.1 8.0 4.1 4.1 6.1 6.1 8.0 8.0 8.0 8.0 8.0 8.0 4.1 4.1 #>  4.1 4.1 8.0 8.0 8.0 8.0 4.1 4.0 4.1 8.0 6.0 8.0 4.1 #> Levels: 4.0 4.1 6.0 6.1 8.0 head(finteraction(mtcars)) # A more crude example.. #>  184.108.40.206.220.127.116.11.18.104.22.168.4.4 #>  22.214.171.124.126.96.36.1995.17.02.0.1.4.4 #>  188.8.131.52.184.108.40.206.220.127.116.11.1.4.1 #>  18.104.22.1688.110.3.08.22.214.171.124.126.96.36.199 #>  188.8.131.520.175.3.15.3.44.17.02.0.0.3.2 #>  184.108.40.206.220.127.116.11.18.104.22.168.0.3.1 #> 32 Levels: 10.4.8.460.215.3.5.422.214.171.124.0.3.4 ... finteraction(mtcars$cyl, mtcars$vs, factor = FALSE) # Returns 'qG', by default unsorted #>  1 1 2 3 4 3 4 2 2 3 3 4 4 4 4 4 4 2 2 2 2 4 4 4 4 2 5 2 4 1 4 2 #> attr(,"N.groups") #>  5 #> attr(,"class") #>  "qG" "na.included" group(mtcars[c("cyl", "vs")]) # Same thing. Use whatever syntax is more convenient #>  1 1 2 3 4 3 4 2 2 3 3 4 4 4 4 4 4 2 2 2 2 4 4 4 4 2 5 2 4 1 4 2 #> attr(,"N.groups") #>  5 #> attr(,"class") #>  "qG" "na.included"