validate 0.9.3 is on CRAN

CRAN just accepted the latest version of our R package validate.

The validate package provides an infrastructure to perform any data quality check in a flexible and extensible way.

This is a minor update with the following new features:

  • New functions exists_any and exists_one to help define cross-record validation rules (thanks to David Salgado)
  • results of sort and aggregate now include key columns (if any)
  • Added JSS paper and CITATION file.

We are also very happy to report that our paper on validate has been accepted by the Journal of Statistical Software. It will take a while before it is published but a preprint was added as a vignette.

  • [PDF] M. van der Loo and E. de Jonge, "Data validation infrastructure for R," Journal of statistical software, p. Accepted for publication, 2019.
    [Bibtex]
    @article{loo2019validation,
    title = {Data Validation Infrastructure for {R}},
    year = {2019},
    author = {MPJ van der Loo and E de Jonge},
    journal = {Journal of Statistical Software},
    pages = {Accepted for publication},
    volume = {},
    note = {},
    pdf = {https://www.markvanderloo.eu/files/statistics/jss3483.pdf}
    }

darth validator

Posted in R, Uncategorized | Tagged , | 2 Comments

Call for abstracts and tutorials: use of R in official statistics 2020 in Vienna

uRos2020

The eight international conference on the Use of R in Official Statistics (#uRos2020) will take place place from 6 to 8 May 2020 at Statistics Austria, the Austrian office of National Statistics.

The meeting in a nutshell

  • 4-5 May: unconfUROS hackathon and General R tutorials for beginners
  • 6 May: Specific R tutorials
  • 7-8 May: uRos2020 Conference

The call for papers is still open.

Keynote speakers: R Core is coming to uRos!

We are very excited to announce that Isabel Molina Peralta and Matthias Templ will give keynotes at uRos2020.

Moreover we are proud to announce that a member of the R core team will join uRos2020 as well! Stay tuned for further announcements.

Contact

Posted in R | Tagged , | Leave a comment

lintools 0.1.3 is on CRAN

Version 0.1.3 of the lintools package was accepted on CRAN today.

This version brings a few internal improvements and switches the testing suite to the tinytest test infrastructure.

lintools is provides basic manipulations of linear systems of equalities and inequalities including: variable elimination (Gaussian elimination, Fourier-Motzkin elimination), Moore-Penrose pseudoinverse, reduction to reduced row echelon form, value substitution, projecting a vector on the convex polytope described by a system of (in)equations, simplifing systems by removing spurious columns and rows and collapsing implied equalities, testing whether a matrix is totally unimodular and computing variable ranges implied by linear (in)equalities.

Posted in programming, R, Uncategorized | Tagged , | Leave a comment

Make your package spherical

Larger packages typically consist of functions that are visible to the users (exported functions) and functions that are used by the exported functions, but that are invisible to the user. For example:


# exported, user-visible function
inch2cm <- function(x){
  x*conversion_factor("inch")
}
# not exported function, package-internal
conversion_factor <- function(unit){
  confac <- c(inch=2.54, pound=1/2.2056)
  confac[unit]
}

We can think of the exported functions (or more correctly, the interface of the exported functins) as the surface of a package, and all the other functions as the volume. The surface is what a user sees, the volume is what the developer sees. The surface is how a user interacts with a package.

If the surface is small (few functions exported, no unnecessary parameters in the interface), users are limited in the ways they can interact with your package, and that means there is less to test. It also means that you, as a package developer, have more room to move and change things in the volume. So as a rule of thumb, it is a good idea to keep the surface small.

Since a sphere has the smallest surface-to-volume ratio possible, I refer to this rule of thumb as as make your package spherical.

Note
This post was first published as a paragraph in the vignette of the tinytest package. I repeat it here with a few changes for more visibility.

Posted in programming, R, Uncategorized | Tagged , | Leave a comment

Checking reverse dependencies: the tiny way

The tools package that comes with base R makes checking reverse dependencies super easy.

  1. Build your package tarball (the pkg_x.y.z.tar.gz file).

    R CMD build /your/package/location

It is a good idea to make sure that the tarball is in a dedicated directory, because the next step will download and install reverse dependencies in the directory where the tarball resides.

  1. In an R terminal type

result <- check_packages_in_dir("/directory/containing/tarball"
                    , reverse = list() )

The result can be printed and summarized and analyzed further if there is any breakage. Here's an example of output when I ran this on my gower package today.


> result
Check results for packages in dir '/home/mark/projects/gower/output':
Package sources: 1, Reverse depends: 5
Use summary() for more information.
> summary(result)
Check results for packages in dir '/home/mark/projects/gower/output':

Check status summary:
                  ERROR NOTE OK
  Source packages     0    0  1
  Reverse depends     1    3  1

Check results summary:
gower ... OK
rdepends_ceterisParibus ... NOTE
* checking dependencies in R code ... NOTE
rdepends_lime ... ERROR
* checking tests ... ERROR
* checking re-building of vignette outputs ... WARNING
rdepends_live ... NOTE
* checking dependencies in R code ... NOTE
rdepends_recipes ... NOTE
* checking dependencies in R code ... NOTE
rdepends_simputation ... OK

(Checking the logs in output/rdepends_lime.Rcheck/00check.log shows that lime fails because of a missing JAVA engine [I just updated my OS and have no JAVA installed yet].)

Notes

  1. Checking reverse dependencies can be done in parallel by setting the Ncpus argument larger than one.
  2. Be aware that the documentation states that (R 3.5.2) This functionality is still experimental: interfaces may change in future versions. Nevertheless, it has worked fine for me so far.
Posted in programming, R | Tagged , , , | Leave a comment

gower 0.2.0 is on CRAN

A new version of R package gower has just been released on CRAN.

Thanks to our new contributor David Turner who was kind enough to provide a pull request, gower now also computes weighted gower distances.

From the NEWS file:

  • gower_dist and gower_topn gain weight argument for weighted matching (thanks to David Turner)
  • gower_dist and gower_topn gain ignore_case argument for automatic column matching.
  • gower_dist now returns numeric(0) invisibly when there are no columns to compare.
  • gower_topn now returns a list with empty matrices when there are no columns to compare.
  • gower_topn now warns when n>nrow(y) and sets n=nrow(y)
  • bugfix: comparing factors with characters would cause a crash (thanks to max Kuhn)

Compute Gower's distance (or similarity) coefficient between records. Compute the top-n matches between records. Core algorithms are executed in parallel on systems supporting OpenMP.

Posted in programming, R | Tagged , , | 1 Comment

uRos2019: tutorials, keynote speakers, registration and call for papers!

The 7th use of R in Official Statistics conference is the event for all things R in the production and use of government statistics. The 7th installment of this conference will take place from 20 to 21 May 2019 at the National Institute of Statistics in Bucharest, Romania.

Keynote Speakers

We are very proud to announce that we have two excellent keynote speakers.

  • Julie Josse will talk about her work on theory and tools related to imputation and inference in the presence of missing data.
  • Giulio Barcaroli will talk about 12 years of using R at ISTAT, the Italian Statistical Office.

Full abstracts can be found here

Tutorials

The conference is preceded by three tutorials on Data Cleaning, Statistical Disclosure Control and Optimal Sampling Stratification.

Call for papers

Yes, abstracts and papers are welcomed until 12 April 2019! You can contribute by sending an abstract in any of the following topics (relating to official statistics):

Sampling and estimation | R in organization | Data cleaning | R in production: data analysis | Methods for official statistics | Shiny applications | Time series | Report and GUI programming | R in production: automation | Big data | Dissemination and visualization

Registration is open

You can now register by following instructions here.

Posted in R | Tagged , , , | Leave a comment

Add a static pdf vignette to an R package

Most vignettes are built when a package is built, but there are occasions where you just want to include a pdf. For example when you want to include a paper. Of course there is a package supporting this, but in this post I will show you how to do it yourself with ease.

The idea is very simple: vignettes can be in LaTeX, and it is possible to include pdf documents in LaTeX using the pdfpages package. So here's the step-by-step recipe:

  1. If you do not already have it, create the vignettes folder in your package directory.
  2. Put your static pdf there. Let's call it mypaper.pdf for now.
  3. Create a .Rnw file with the following content.
\documentclass{article}
\usepackage{pdfpages}
%\VignetteIndexEntry{author2019mypaper}

\begin{document}
\includepdf[pages=-, fitpaper=true]{mypaper.pdf}
\end{document}

That's it.

Some notes.

  1. This repo contains an example.
  2. The option fitpaper=true is necessary because the Sweave package that is included when the vignette is built somehow causes the pages to rescale if it is not included.
  3. If you post your package to CRAN, myfile.pdf will be deleted from the directory so it is not part of a binary download.
  4. You can include errata or other notes, for example as follows:
\documentclass{article}
\usepackage{pdfpages}
%\VignetteIndexEntry{author2019mypaper}

\begin{document}
\includepdf[pages=-, fitpaper=true]{mypaper.pdf}

\newpage{}
\subsection*{Errata}

A few things were borked in the original publication, here
is a list of sto0pid things I did:

\begin{itemize}
\item{fubar 1}
\item{fubar 2}
\end{itemize}

\end{document}
Posted in programming, R | Tagged , , | 1 Comment

The program for uRos2018 is online

The uRos2018 conference is aimed at professionals and academics who are involved in producing or consuming official (government) statistics.

We are happy to announce that we recently posted the full program of the 6th international conference on the use of R in official Statistics (uRos2018) on our website.

In summary:

  • Six tutorials in the areas of
    • Data Cleaning
    • Network Analyses
    • Survey Estimation
    • Data manipulation with data.table
    • Analyzing spatial data
    • Visualizing spatial data.
  • Two keynote speakers:
    • Alina Matei, professor of statistics at the University of Neuchatel and maintainer of the sampling package.
    • Jeroen Ooms, R superprogrammer and maintainer of R and Rtools for Windows (UC Berkeley)
  • Eleven sessions with contributed talks with five presentations from
    all over the world.
  • One session devoted to the results of a two-day unconf that is held prior to the conference.
  • One social dinner 🙂
  • Two journals will devote a special topic to the conference.

All the abstracts will be published online soon.

Registration is still open

  • You are welcome to register
  • Follow us on twitter for the latest news and updates!
Posted in programming, R | Tagged | Leave a comment

stringdist 0.9.5.1: now with C API

Version 0.9.5.1 of stringdist is on CRAN. The main new feature, with a huge thanks to our awesome new contributor Chris Muir, is that we made it easy to call stringdist functionality from your package's C or C++ code.

The main steps to get it done are:

  1. Make sure to add stringdist to the Imports: and LinkingTo: fields in your DESRIPTION file
  2. Add the #include <stringdist_api> to your C/C++ source file.
  3. Start using stringdist from C!

Here's an example source file

#include <R.h>
#include <Rdefines.h>
#include <stringdist_api.h>


SEXP my_soundex(SEXP strings, SEXP useBytes){
  Rprintf("\nWow, using 'stringdist' soundex encoding, from my own C code!\n");
  return sd_soundex(strings, useBytes);
}

Great! how can I learn more?

  • The full API is desribed in a pdf file that is generated from doxygen that comes with the package. You can find it by typing ?stringdist_api on the R command line.
  • A minimal example package that links to stringdist is available on GitHub
  • A more sophisticated package with more elaborate examples can be found here: refinr (By Chris)

Any other news?

A few fixes, and a couple of long-deprecated function arguments have finally been removed. Check out the NEWS file on CRAN for a complete overview.

Happy coding!

Posted in programming, R | Tagged , | 2 Comments