seplyr
is an R
package that makes it easy to program over dplyr
0.7.*
.
To illustrate this we will work an example.
Suppose you had worked out a dplyr
pipeline that performed an analysis you were interested in. For an example we could take something similar to one of the examples from the dplyr
0.7.0
announcement.
suppressPackageStartupMessages(library("dplyr"))
packageVersion("dplyr")
## [1] '0.7.2'
cat(colnames(starwars), sep='n')
## name
## height
## mass
## hair_color
## skin_color
## eye_color
## birth_year
## gender
## homeworld
## species
## films
## vehicles
## starships
starwars %>%
group_by(homeworld) %>%
summarise(mean_height =
mean(height, na.rm = TRUE),
mean_mass =
mean(mass, na.rm = TRUE),
count = n())
## # A tibble: 49 x 4
## homeworld mean_height mean_mass count
## <chr> <dbl> <dbl> <int>
## 1 Alderaan 176.3333 64.0 3
## 2 Aleen Minor 79.0000 15.0 1
## 3 Bespin 175.0000 79.0 1
## 4 Bestine IV 180.0000 110.0 1
## 5 Cato Neimoidia 191.0000 90.0 1
## 6 Cerea 198.0000 82.0 1
## 7 Champala 196.0000 NaN 1
## 8 Chandrila 150.0000 NaN 1
## 9 Concord Dawn 183.0000 79.0 1
## 10 Corellia 175.0000 78.5 2
## # ... with 39 more rows
The above is colloquially called "an interactive script." The name comes from the fact that we use names of variables (such as "homeworld
") that would only be known from looking at the data directly in the analysis code. Only somebody interacting with the data could write such a script (hence the name).
It has long been considered a point of discomfort to convert such an interactive dplyr
pipeline into a re-usable script or function. That is a script or function that specifies column names in some parametric or re-usable fashion. Roughly it means the names of the data columns are not yet known when we are writing the code (and this is what makes the code re-usable).
This inessential (or conquerable) difficulty is largely a due to the preference for non-standard evaluation interfaces (that is interfaces that capture and inspect un-evaluated expressions from their calling interface) in the design dplyr
.
seplyr
is a dplyr
adapter layer that prefers "slightly clunkier" standard interfaces (or referentially transparent interfaces), which are actually very powerful and can be used to some advantage.
The above description and comparisons can come off as needlessly broad and painfully abstract. Things are much clearer if we move away from theory and return to our practical example.
Let’s translate the above example into a re-usable function in small (easy) stages. First translate the interactive script from dplyr
notation into seplyr
notation. This step is a pure re-factoring, we are changing the code without changing its observable external behavior.
The translation is mechanical in that it is mostly using seplyr
documentation as a lookup table. What you have to do is:
- Change
dplyr
verbs to their matchingseplyr
"*_se()
" adapters. - Add quote marks around names and expressions.
- Convert sequences of expressions (such as in the
summarize()
) to explicit vectors by adding the "c()
" notation. - Replace "
=
" in expressions with ":=
".
Our converted code looks like the following.
library("seplyr")
starwars %>%
group_by_se("homeworld") %>%
summarize_se(c("mean_height" :=
"mean(height, na.rm = TRUE)",
"mean_mass" :=
"mean(mass, na.rm = TRUE)",
"count" := "n()"))
## # A tibble: 49 x 4
## homeworld mean_height mean_mass count
## <chr> <dbl> <dbl> <int>
## 1 Alderaan 176.3333 64.0 3
## 2 Aleen Minor 79.0000 15.0 1
## 3 Bespin 175.0000 79.0 1
## 4 Bestine IV 180.0000 110.0 1
## 5 Cato Neimoidia 191.0000 90.0 1
## 6 Cerea 198.0000 82.0 1
## 7 Champala 196.0000 NaN 1
## 8 Chandrila 150.0000 NaN 1
## 9 Concord Dawn 183.0000 79.0 1
## 10 Corellia 175.0000 78.5 2
## # ... with 39 more rows
This code works the same as the original dplyr
code. Obviously at this point all we have done is: worked to make the code a bit less pleasant looking. We have yet to see any benefit from this conversion (though we can turn this on its head and say all the original dplyr
notation is saving us is from having to write a few quote marks).
The benefit is: this new code can very easily be parameterized and wrapped in a re-usable function. In fact it is now simpler to do than to describe.
For example: suppose (as in the original example) we want to create a function that lets us choose the grouping variable? This is now easy, we copy the code into a function and replace the explicit value "homeworld"
with a variable:
starwars_mean <- function(my_var) {
starwars %>%
group_by_se(my_var) %>%
summarize_se(c("mean_height" :=
"mean(height, na.rm = TRUE)",
"mean_mass" :=
"mean(mass, na.rm = TRUE)",
"count" := "n()"))
}
starwars_mean("hair_color")
## # A tibble: 13 x 4
## hair_color mean_height mean_mass count
## <chr> <dbl> <dbl> <int>
## 1 auburn 150.0000 NaN 1
## 2 auburn, grey 180.0000 NaN 1
## 3 auburn, white 182.0000 77.00000 1
## 4 black 174.3333 73.05714 13
## 5 blond 176.6667 80.50000 3
## 6 blonde 168.0000 55.00000 1
## 7 brown 175.2667 79.27273 18
## 8 brown, grey 178.0000 120.00000 1
## 9 grey 170.0000 75.00000 1
## 10 none 180.8889 78.51852 37
## 11 unknown NaN NaN 1
## 12 white 156.0000 59.66667 4
## 13 <NA> 141.6000 314.20000 5
In seplyr
programming is easy (just replace values with variables). For example we can make a completely generic re-usable "grouped mean" function using R
‘s paste()
function to build up expressions.
grouped_mean <- function(data,
grouping_variables,
value_variables) {
result_names <- paste0("mean_",
value_variables)
expressions <- paste0("mean(",
value_variables,
", na.rm = TRUE)")
calculation <- result_names := expressions
print(as.list(calculation)) # print for demo
data %>%
group_by_se(grouping_variables) %>%
summarize_se(c(calculation,
"count" := "n()"))
}
starwars %>%
grouped_mean(grouping_variables = "eye_color",
value_variables = c("mass", "birth_year"))
## $mean_mass
## [1] "mean(mass, na.rm = TRUE)"
##
## $mean_birth_year
## [1] "mean(birth_year, na.rm = TRUE)"
## # A tibble: 15 x 4
## eye_color mean_mass mean_birth_year count
## <chr> <dbl> <dbl> <int>
## 1 black 76.28571 33.00000 10
## 2 blue 86.51667 67.06923 19
## 3 blue-gray 77.00000 57.00000 1
## 4 brown 66.09231 108.96429 21
## 5 dark NaN NaN 1
## 6 gold NaN NaN 1
## 7 green, yellow 159.00000 NaN 1
## 8 hazel 66.00000 34.50000 3
## 9 orange 282.33333 231.00000 8
## 10 pink NaN NaN 1
## 11 red 81.40000 33.66667 5
## 12 red, blue NaN NaN 1
## 13 unknown 31.50000 NaN 3
## 14 white 48.00000 NaN 1
## 15 yellow 81.11111 76.38000 11
The only part that requires more study and practice was messing around with the expressions using paste()
(for more details on the string manipulation please try "help(paste)
"). Notice also we used the ":=
" operator to bind the list of desired result names to the matching calculations (please see "help(named_map_builder)
" for more details).
The point is: we did not have to bring in (or study) any deep-theory or heavy-weight tools such as rlang
/tidyeval
or lazyeval
to complete our programming task. Once you are in seplyr
notation, changes are very easy. You can separate translating into seplyr
notation from the work of designing your wrapper function (breaking your programming work into smaller easier to understand steps).
The seplyr
method is simple, easy to teach, and powerful. The package contains a number of worked examples both in help()
and vignette(package='seplyr')
documentation.
Categories: Coding Opinion Tutorials
jmount
Data Scientist and trainer at Win Vector LLC. One of the authors of Practical Data Science with R.
So what is “
:=
“? It turns out it is a reserved symbol for an old Pascal-style assignment operator that is no longer bound to an implementation in currentR
.This means it is unique among
R
operators in that:It doesn’t have a base-R implementation (meaning it is somewhat up for grabs).
It doesn’t need
%%
notation like most user defined operators.It binds late as you would expect an assignment operator to (meaning we don’t need parenthesis in many situations).
Below is some example code playing with the
:=
symbol.Now some packages do already use the
:=
operator, but it isn’t something anyone can really claim to own or reserve. Here are the packages I am aware of using:=
:data.table
.data.table
does export an implementation of:=
. However, I believe it is largely used in adata.table
controlled context, soR
package semantics should ensuredata.table
does not get clobbered byseplyr
.rlang
/tidyeval
.rlang
/tidyeval
assigns:=
to be~
, which seems like a waste as~
does in fact exist and few users should be importingrlang
/tidyeval
directly as it is a toolbox largely used to build other packages.rlang
/tidyeval
also uses the fact that expressions that treat:=
as an assignment are not parse-errors, allowing it to capture unevaluated user expressions with the:=
symbol in an assignment position. Sincerlang
/tidyeval
is essentially running its own interpreter (aseval
andapply
have been called the fundamental equations of functional languages, so when you override them you essentially have a new interpreter or language) we expect our mere binding of:=
can’t interfere withrlang
/tidyeval
operations.dplyr
.dplyr
does not export a implementation for:=
but uses it to allow specification of left-hand-sides of assignment in unparsed user expressions (especially indplyr::mutate
anddplyr::summarize
).Example of
data.table
correct use of:=
even after loadingseplyr
:Example of
dplyr
use of:=
, notice how to the user the effect seems very similar to theseplyr
use.Very similar
seplyr
code:Now, finally what is
seplyr::`:=`
? It is so short I think it can be considered elegant:(More documentation in
help(":=", package="seplyr")
.)The above is something I am really trying to strive for in my packages: working with
R
(instead of overriding or replacing large swaths of it). Thewrapr
dot-pipe operator is similarly concise (please seehelp("%.>%", package="wrapr")
for details).Developers were very happy with base R and plyr. No one wanted the Jekyll/Hide NSE/SE complication. Now with dplyr you have a choice of not using it, but the NSE/SE confusion is going to be added to ggplot and others. Are you going to make seplyr like variations for those packages as well? I doubt that.
NSE/SE is going to make ggplot lot more difficult to use and understand.
Obviously I am not part of the
tidyverse
team and am not privy to their plans.But that being said.
I think adapting
ggplot2
torlang
/tidyeval
is not going to look as good as thedplyr
adaption from the user point of view. There are often a lot more moving parts in aggplot2
plot (multipledata.frame
s, aesthetics, statistics, facets, colors, groups, legends, keys, and compositing by name) than in a typicaldplyr
pipeline (which tends to be more a linear structure with data entering at one end). I am in fact worried it will break the user facing plotting interface and further worried that the usefulaes_string()
method will be deprecated to force users to try the new interface (remember, thedplyr::"underscore"
methods ended up deprecated).I don’t anticipate a very graceful way re-adapt what I assume is coming back to standard interfaces. I expect in production code I will pretend I have known column names either by wrapping everything in one giant
wrapr::let()
block or by using transient column re-namings viareplyr::replyr_apply_f_mapped()
(though this requires separate control of the axes names).It is kind of too bad this is the next announced
ggplot2
update.ggplot2
is ahead on notation, butbokeh
,plotly
, and others are getting ahead on a lot of important rendering and interaction features (i.e. plots are not all for the programmer).I am trying to shrink the visible footprint of
seplyr
. To that end the development version:No longer loads
dplyr
into the user namespace (so if you wantdplyr
names available you must calllibrary("dplyr")
directly even if you have already calledlibrary("seplyr")
).Declares the operator “
:=
” in the usualS3
manner and now only supplies implementations for a few appropriate types (string vectors, name vectors, and lists). The idea is: if all packages wanting to use:=
did this then there would have even less of a chance of packages interfering with each other. To see the implementation you now typeprint(named_map_builder)
instead ofprint(`:=`)
.Hi, isn’t get() function combined with dplyr sufficient enough to achieve the desired re-usability? Is there other reason why seplyr is necessary?
Thanks for the comment, I appreciate people sharing their views and experience.
I don’t consider
seplyr
necessary. What I consider it is: a proof by example that standard (value oriented) interfaces are in fact sufficient for graceful data manipulation. The unstated idea is: it would have been possible to have something likedplyr
without a lot of the other issues. I know we don’t, but I wanted to remind people it was possible.Or (with a sympathetic eye): suppose instead of translating the pipeline from
dplyr
notation toseplyr
notation (which puts one in the silly “don’t like the metric system because I have to use math to convert that the US is stuck in”) one had just written theseplyr
pipeline in the first place. My point is: it isn’t too much worse (anddplyr
itself needs the:=
notation if one want to substitute on left hand sides of expressions).Do you mind explaining your point with code?
Thanks
My blogging comment system isn’t the best- so it may not be clear of Joe is asking a question of me or a question of Sungmin (I think the admin panel says it is a question to Sungmin). That being said I think we are not all of one opinion on this, which I want to respect.
Hi John,
Yes, it is a question for Sungmin’s observation but you can also expand on it. I do use `wrapr::let()` quite often but it’d be nice to see an application with `get()` for comparison purposes.
My group’s preference remains
dplyr
withwrapr::let()
doing all of the re-mapping (as shown here).I don’t quite see how
base::get()
works in this context and am beginning to wonder if it might be a typo (forwrapr::let()
). If not I’d love to see an example, as it would be a neat new idea. If so then it sounds like all three of use may be closer in opinion than any of us anticipated.It isn’t so much that
dplyr
can’t perform all of these tasks withrlang
/tidyeval
and thedplyr::*_at()
methods, it is just each one works slightly differently (for instancedplyr::rename_at()
) and they are not strict (they look for values more places than I would like- hiding potential errors, which I will indicate below). I feel part of the issue isrlang
/tidyeval
as described is a different creature thanrlang
/tidyeval
as implemented (so it is easy for the described version to appear better).Here is a example closer to my point where it isn’t clear that the
seplyr
is really worse than thetidyverse
version.Inaki has posted an article Programming with dplyr by using dplyr https://www.enchufa2.es/archives/programming-with-dplyr-by-using-dplyr.html where he makes a case that seplyr isn’t needed.
Thanks for the link, looks like a pretty good analysis. I don’t agree, but some of that is a matter of opinion. I would say if
dplyr
/rlang
/tidyeval
were a bit more regular (worked more like as described, and had fewer exceptional cases) I would be closer to agreeing (here are some of my notes on corner cases).