Steve Pousty | CrunchyData Blog

Using PostgreSQL to Shape and Prepare Scientific Data

Steve.Pousty@crunchydata.com (Steve Pousty) — Fri, 19 Feb 2021 12:02:22 EST

Today we are going to walk through some of the preliminary data shaping steps in data science using SQL in Postgres. I have a long history of working in data science, including my Masters Degree (in Forestry) and Ph.D. (in Ecology) and during this work I would often get raw data files that I had to get into shape to run analysis.

Whenever you start to do something new there is always some uncomfortableness. That “why is this so hard” feeling often stops me from trying something new, but not this time! I attempted to do most of my data prep and statistical analysis within PostgreSQL. I didn’t allow myself any copy and paste in google sheets nor could I use Python for scripting. I know this is hard core and in the end may not be the “best” way to do this. But if I didn’t force myself to do it, I would never understand what it could be like.

The Project

Next I needed to pick a modeling project. We have a few interesting data sets in the Crunchy Data Demo repository but nothing that looked good for statistical modeling. I thought about trying to find something a little more relevant to my life.

Photo by keppet on flickr

You may have noticed that we are having a few fires here in Northern California. Quite a few Crunchy folks live here. I'm in Santa Cruz (with a fire that was less than 7 miles away), Craig lives near Berkeley, Daniel and Will live in SF. We've compared notes on the sky (fun pic here), the amount of ash on cars, and PurpleAir values.

I thought it would be interesting to see if logistic regression could predict the probability of a fire based on the historical weather and fire patterns. All the details for how we obtained historical California fire data is here and here is the weather data.

Here is a preview of the fire data:

And here is a preview of the weather data:

Initial Import of Data

You can see on the pages above some of the initial cleaning we did. Most of it is minor though crucial things like renaming columns, taking a separate date and time column, and combining them into a timestamp, or changing the value on an obvious typo.

Let’s dig into the concatenating the date and time into a timestamp for the weather data (step 5). The old me would have put all the data in a spreadsheet and then used a concatenate function to put the two together. I would do one cell with the formula and then copy and paste into all the cells.

Instead, this is all we need:

UPDATE weather SET date_time = to_timestamp((date || ' ' || time || '-7'),
'YYYYMMDD HH24:MI');

One of the obvious advantages you should see right away is that we only have one line of code to accomplish what would take several manual steps. Since it’s in code if we need to reimport the data or modify it for another data set, we just have to reuse this line. We could even use this as part of an automated script.

The other thing you should notice is that we were able to actually add a timezone to the data (the -7). And finally, we were able to act on the data as a whole unit. That means we do the operation all at once without fat-fingering the data or forgetting to paste some sort of rows. The data stays together and all the operations happen at “the same time”. If we wanted to be even more careful we could have wrapped the statement in a transaction.

Subsetting Our Data

The first thing we are going to do to shape our data is subset the fires. Our weather station is in Northern California and so it probably doesn’t make sense to try and predict Southern California fire probability given weather in the north. This is again a one liner to subset our data:

SELECT fire19.* INTO ncalfire FROM fire19, fire19_region WHERE
*st_covers\*(fire19_region.geom, fire19.geom) AND fire19_region.region =
'Northern';

We selected all the columns from the original data where the Northern Region covers all the fire data. We use ST_Covers rather than ST_Contains because of some unexpected behaviour in ST_Contains. By using st_covers, our operation will not include any fires whose boundaries go outside the Northern Region, for example if the fire spread into Oregon.

Getting ready for Logistic Regression

Logistic regression needs the response variable (did a fire happen on that day), be coded as a 1 = fire happened, 0 = no fire. When I started to examine the data, as I should have known based on living in California, multiple fires can start on the same day. So we need to aggregate data from multiple days into a single entry. With SQL we don’t have to lose any information. Here is the one line that does what we need:

WITH grouped_fire AS (
 SELECT alarm_date, count(*) as numfires, string_agg(fire_name, ', ') AS names,

*st_collect*(geom)::geometry(geometrycollection, 3310) AS geom FROM ncalfire
GROUP BY alarm_date
 )
SELECT w.*, grouped_fire.\*, 1 AS hasfire INTO fire_weather FROM weather w, grouped_fire
WHERE grouped_fire.alarm_date = w.date_time::date;

So the first part of this query starting WITH grouped_fires until the ) is called a CTE or common table expression. We do all our aggregation in the CTE based on alarm_date (specified by the GROUP BY alarm_date). Our output will be for each alarm date:

Return the number of fires that happened on that day.
Aggregated all the strings for the fire names into one string separated by “,” (we could have also used an an array of string).
Collect the geometries of all the different fires on that day into one geometry collection.

Now we can use the information from the CTE like a table in the second part of the query. In the next part we grab all the columns from the CTE along with all the columns in the original table and then we just put a 1 in for all entries in an output column named hasfire.

As a side note, I am a big fan of bringing clarity to boolean columns by prefixing them with is or has so you know what the true condition means.

Then we join the CTE date to the original table by matching dates. We truncate the timestamp column by casting it to a date. Notice we are selecting into a new table and the output of this command will actually go into a table rather than showing on the screen. With that we now have a table with all the weather data, dates of fires, and a 1 in the hasfire column.

Fixing our Geometry

You will notice above we had to make a geometrycollection to group the fires geometries together. Most desktop and other GIS software don’t know what to do with a geometrycollection geometry type. To fix this problem we are going to cast the geometries to multipolygons. A multipolygon allows a single row to have multiple polygons. The easiest example of this is a town or property that includes islands. A multipolygon allows the town, along with its islands, to be seen as a single entry in the table. Here is the call we use:

ALTER TABLE fire_weather ALTER COLUMN geom type geometry(multipolygon,3310)
USING st_collectionextract(geom, 3);

ST_collectionextract takes a multigeometry and collects all the geometries you ask for and returns a multigeometry of the requested type. The 3 in st_collectionextract is the “magic number” meaning polygons

Creating our non-fire data

We created our fire data and fixed it’s geometry. Now we have to go back to the weather data and assign 0 to all the days where there was no fire:

WITH non*fire_weather AS (
SELECT weather.* FROM weather WHERE id NOT IN (SELECT id FROM fire*weather)
)
SELECT non_fire_weather.*, null::date AS alarm_date, 0::bigint AS numfires,
null::text AS names, null::geometry(MultiPolygon,3310) AS geom, 0 AS hasfire
INTO non_fire_weather FROM non_fire_weather;

Again we are going to start with a CTE but this time the first query is used to find all the days that are NOT in the new fire + weather table above. This gives us a “table” with only weather data on days where a fire did not occur. Then in the second part of the query we add in all the columns that we added to our fire_weather table. Finally we put the results into a new table called non_fire_weather.

Now we combine the fire_weather and non_fire weather tables into one master table called alldata:

SELECT _ INTO alldata FROM non_fire_weather UNION SELECT _ FROM fire_weather;

We did this merging with the full data set in case we want to do other types of analyses with this data. For example, we actually have counts of fire on days and we also have total area of fire burned (from the polygons). These variables might lead to other interesting analysis in the future.

Wrap Up

Let’s wrap up and look at all the good work we were able to do using SQL for data science. We took some tables imported from raw data, turned dates and time into more usable formats, aggregated the fire events, kept the geometries intact, created a code for event versus not event, and created a large master data set.

All of this was done with lines of code, no manual steps. Doing our data shaping this way makes our process easily repeatable if we need to do it again. We could actually put it in a script and automate the importing of new data. On a personal note, it makes my life so easy if I need to create a new version of the dataset or import the data to a new server.

I hope you found this code and the examples useful in your data science work (here is the github repo for it). What is your experience with using SQL to shape your data before doing analysis? Do you have some tips or tricks you would like to share? Leave us comment on the Crunchy Data twitter account. Have fun with your analysis and code on!

Cover image By Annette Spithoven, NL

R Predictive Analytics in Data Science Work using PostgreSQL

Steve.Pousty@crunchydata.com (Steve Pousty) — Tue, 12 Jan 2021 04:00:00 EST

Greetings friends! We have come to our final blog post in my series about the data science workflow using PostgreSQL. In the last blog post, we used PL/R to create a function which returns the output from a logistic regression model trained on our fire data. We then took that model object and stored it into a separate table.

Today we are going to finish up by showing how to use that stored model to make predictions on new data. By the way, I did all of the Postgres work for the entire blog series in Crunchy Bridge. I wanted to focus on the data and code and not on how to run PostgreSQL.

Numer models

When people make numer models they usually have one of two purposes in mind.

Using the model to get a better understanding of the processes that leads to the outcome.
Trying to get the most accurate prediction of the outcome.

These two options can sometimes be at odds in your analysis.

You can usually make a more accurate prediction model using black box techniques such as neural networks or generalized additive models. But these techniques do not allow any inference about magnitude or directions of cause and effect. There are other reasons they can be at odds, such as predictive power versus parsimony of explanatory variables, but we are not going to go in depth on that today.

In our example, we are not particularly interested in an explanatory model since we have little ability to control the weather. Instead, we are more interested in a predictive model to give us fire occurrence prediction as accurately as possible.

The full use case for using this model would be:

Run a weather forecast model predicting tomorrow’s weather for the explanatory variables used in our logistic regression.
Plug those weather values into our logistic regression model (covered in today's blog post).
Return the predicted probability of tomorrow’s weather.
Give this information to someone who can use it to make better decisions.

Our use case is very similar to other non-fire use cases. Here are some examples:

Create a credit card fraud model, store it in the database, write a trigger on insert that predicts the probability of fraud.
Create a model that predicts market demand for an item, store it in the database, produce a report every morning that shows store managers how much inventory to order.
Build a predictive model of your favorite football team (yes I mean the U.S. version and the International version), update based on stats from their last game/match, predict their likelihood of winning their next match.

Let’s get into the final step on our data science journey.

Using the R model to predict fire probability

We are going to use PL/R to do the statistical work, just like we did in the last post. Unless you know R and PL/R, I highly recommend you go back and read it. I am not going to cover the benefits of PL/R or the pieces of setting up a PL/R function.

At the end of last post we took the R object for a logistic regression model and stored it in a bytea column along with some metadata columns (including a timestamp column for record insertion).

The new function

The prize we get for using R and storing the model object in a table is the ability to write a simple function to generate a prediction. It’s so small I am going to paste the entire function here:

CREATE OR REPLACE FUNCTION final.predict_logistic( model bytea, newdata
final.test, out probability numeric)

as $$

probability <- predict(model, data.frame(newdata), type = 'response')

return(probability)

$$
LANGUAGE 'plr';

We are creating this function in the final schema. The bytea we are passing in will be the R model we stored in the table.

The declaration for the newdata parameter might be new for some people. If you recall, the test table contains data we separated out from the training data set and has the same schema as the data table. By declaring it of type final.test we are indicating that we expect a record to be passed that matches the schema of the table in the final schema named test.

Finally we declare that we are returning, via an out variable, the calculated probability.

One line of R code

The actual code of our function is 1 LINE! I could have returned the call to predict, but I like separating those calls out for clarity.

This line also shows why R is such a powerful language for statistics and data science. Built-in are core functions for most of the methods data scientists and statisticians use. It is a Domain Specific Language for statistics and data science.

Our call to predict:

Takes a R model created with a linear or generalized linear model (our stored model).
Takes a data frame which contains one or more rows of data that matches the independent variables used in the model.
We then tell the function we want the predicted response rather than the standard errors on the response.
The function returns a value between 0 and 1, representing the probability of fire.

And that’s it!

Our function does all we need it to do. The data scientist gives this function to the DBA and they put it in the database. Now, application developers can just pass in data and get back a prediction of fire probability and the data scientist can sleep soundly because nobody can “mess up” implementing a prediction function.

Calling our function in SQL

Using this function is quite simple since we only need to pass in 2 parameters.

SELECT final.predict_logistic(

newdata := (SELECT f.\*::final.test FROM final.test AS f ORDER BY f.date_time
limit 1),

model := (SELECT model AS the model FROM final.logistic_models AS l ORDER BY
l.created_at DESC LIMIT 1) ) AS fire_probability;

Passing in the new explanatory data we do a select on final.test.* and cast it to final.test like the function expects. To limit the rows passed in, submit the oldest data value.

Pass in the R model, we select the model column from the final.logistic_models table. Again, we sort the results but this time we only return the newest model. Voila!

To make this easier for application developers to use, you might want to make this query into a view.

Wrapping up the whole series

With this post we have come to the conclusion of our series on using Postgres and SQL in a data science/research workflow.

We covered all these topics without using any code running outside of Postgres:

Cleaning up the data
Massaging the data to fit our needs
Subsetting the data
Combining two data sets to create the data we needed for analysis
Randomly sampled our table into training and testing data
Created a function to create new columns that contained the centered and standardized values for the explanatory variables
Created another function to fit a logistic regression model with R using the centered and standardized data
Stored the R logistic regression model in a new table
Finally, wrote a function that accepts the stored model and new data to predict the probability of a new fire

If I had told you, before this series, that we were going to do all this work inside PostgreSQL, I bet the response for most of you would have been “no way!”. And that was the point of this series.

Like my coworker Craig said:

“You probably won’t get them all the way there, but you will have opened their world to see what is possible.”

One of the messages I hope I imparted to you was the clear separation of concerns you can achieve by using PostgreSQL as the central node in your architecture.

The data can be cleaned and duplicated with strong permission handling. The data scientists can work with the db data in their native environment to come up with the algorithms. The DBA or sysadmin can then put those algorithms into the database as functions available to application developers.

Finally the application developer gets the benefit of not having to rewrite the algorithm in their language, instead they can just do a simple SQL call. Every different “user” of the database gets to interact with the database for their needs without reaching into other “users” domains.

One other benefit of using the approach outlined here is that all these steps can be put in a text file and run against the database. You don't have to go back and copy and paste to recreate data in a spreadsheet. All your steps are clearly laid out in code that can be rerun anytime you need.

My final goal was to show how learning more SQL would lead to big benefits in your day to day work. I was pleasantly surprised by the functionality I never knew existed in PostgreSQL, like randomly sampling a table.

Did this series make you think more about how you use PostgreSQL? Have you used PostgreSQL in data science workflows? I would love to hear more about your experience and insights by sending messages to our twitter account.

Happy data sleuthing!

Using R in Postgres for Logistic Regression Modeling

Steve.Pousty@crunchydata.com (Steve Pousty) — Wed, 06 Jan 2021 04:00:00 EST

Greetings friends! We have finally come to the point in the Postgres for Data Science series where we are not doing data preparation. Today we are going to do modeling and prediction of fire occurrence given weather parameters… IN OUR DATABASE!

Quick recap:

We found some data on historical fires and historical weather data for California.
We fixed up the data and merged it to make it more appropriate for logistic regression.
We randomly sampled non-fire data and split the data into training and verification data sets.
We put our final data for analysis into a PostgreSQL schema named “final”.
Finally, we standardized and centered our weather parameters in new columns in our table.

For those of you not in statistics or data science, this may seem like a lot of work just to run a logistic regression. But as people in those fields will tell you, cleaning, shaping, and transforming your data for analysis is actually where you spend the majority of your time.

Statistical modelling with PL/R

For those who have been living under a rock and haven’t heard of R, it is a open-source programming language focused on statistics and data science with a HUGE ecosystem behind it. While it may look strange to you - it’s extremely well written for matrix calculations and statistical analysis.

As we have seen before, PostgreSQL has the ability to write functions in different programming languages. My colleague at Crunchy Data, Joe Conway, did just that for R in an embedded language PL/R. I don’t have time or space to teach PL/R in a blog post but there are a lot of good materials, including a hands-on tutorial and presentations. Today I will mostly show the flow a data scientist can use and some more interesting properties of PL/R.

Getting data from PostgreSQL in and out of R

One of the best properties Joe built into the language is RPostgresql compatibility support. If you work in R, the driver you use to connect to PostgreSQL is RPostgresql. What Joe did was stub out all the RPostgresql connection methods in PL/R to become noops functions. Instead, PL/R assumes it is connecting to the current database. As the documentation says:

These functions nominally work like their RPostgreSQL counterparts except that all queries are performed in the current database. Therefore all driver and connection related parameters are ignored, and dbDriver, dbConnect, dbDisconnect, and dbUnloadDriver are no-ops.

The implication of this for you, dear reader, is that you can write your normal RPostgresql R code on your desktop, and then almost drop it straight into PL/R function. Brilliant!

This pattern above is exactly what I did for my function development. I used my favorite (and best) IDE in the whole world, IDEA, with its R plugin to do all my R work on my desktop. I just used RPostgresql to connect to the DB where I would be creating my function.

Creating the logistic regression model

Since my main purpose was to show the use of PostgreSQL in a data science/statistical analysis workflow, I did not go through the normal model fitting techniques. Instead, I looked at the independent variables that would seem related to fire risk:

Minimum relative humidity
Maximum air temperature
Precipitation
Solar Radiation

The parameters and overall model were all significant at p < 0.05 so I deemed it “good enough”.

Onward and upward!

Putting the model into PL/R

The next phase is to take the model off my desktop and put it on the Postgres server. Given the work Joe did before on RPostgresql compatibility, I made 0 changes to my R code when creating the PL/R function. Let’s go through the steps I did to create the function.

The first step was to add the PL/R extension to my database. I need to do this command as a superuser (typically the user “postgres”). I will also need to add the function as a superuser since PL/R is an untrusted language. Note, this tutorial actually teaches you how to make PL/R a trusted language, but please don’t do that on your production DBs unless you know what you are doing.

CREATE EXTENSION plr;

And with that we are ready to add our function.

Turn my R code into a Postgres function

The code for the function is a standard template for a function (if you are not familiar with functions in PostgreSQL, here is a nice tutorial). Here is the complete code for my embedded function. In this blog post I am going to break it down by topical areas.

Function Signature

The function signature is a typical Postgres function signature.

CREATE OR REPLACE FUNCTION final.logistic_model()

RETURNS bytea as $$

We create this function in the final schema, which is also where the data is located.
We are not passing in any data because we will get the table data through the R code.
We are going to return bytea because we are actually going to store the logistic model created in R into another table.

R PostgreSQL Connection and Data Code

To begin our R function we need to get our data out of the table and into R. Remember, all of this connection and data loading code was just from me working in R on my desktop. The PL/R extension will actually “translate” all of these calls into direct db calls.

We load the library for the R, RPostgreSQL, which will handle all the connections and queries.
The next two lines load the driver and make the connection to our Postgres instance.
The following line uses the connection (con) to do a SQL query against our database and put the results in a R dataframe (df).

So once we make this into PL/R none of these connection calls actually use RPostgreSQL, they use the PL/R versions which are no-ops. And it executes the dbGetQuery against the current database.

Executing R Code

Other than the data query, our function only executes one line of R code. We use the standard way to do logistic regression in R.

logmodel_solar <- glm(hasfire ~ cs_rh_min + cs_air_max_temp + cs_precip + cs_solar, data=df, family = binomial("logit"))

Use the glm function to carry out a generalized linear model.
hasfire is the dependent variable (coded as 1 for fire that day and 0 for no fire)
The dependent variable is going to be linearly related to 4 independent variables
Use the dataframe, df, as the source of all the data values
Use the link function of type binomial logit. Using this link type changes the predicted values from any real number in ordinary linear regression, to the 1,0 in logistic regression.
Save the output into an object named logmodel_solar.

Included in logmodel_solar is not only the significance of the overall model but the actual parameter values for each independent variable. This will be important later in the blog.

Cleaning Up and Returning

The next two lines release the connection to the database (a no-op in PL/R) and unload the driver. Finally, we return logmodel_solar. Now you may be wondering, what happens to an R object when we pass it back to PostgreSQL. The PL/R doc clearly lays out the translation between R and PostgreSQL data types. In our case, since we are returning an R object, PL/R will convert that to a bytea type in PostgreSQL.

How to Use the Function

With our spiffy function in hand, how do we use it? Well it’s as simple as a SQL call. We are actually going to combine running the function with saving its model to a table. Our table for holding the function output only has the following structure:

final.logistic_models

id: self-incrementing integer column to act as the primary key
name: text field that can hold a name for the model
model: bytea column to hold the model object
created_at: timestamp column which defaults to now()

So running our logistic regression model and creating a new table record is as simple as:

INSERT INTO final.logistic_models (name, model) VALUES ('first run', final.logistic_model());

With that in place, we could actually run a job every week to refit the logistic regression model based on newly input data. As a data scientist I can pull the more recent model back out of the database and look at the model object in R on my desktop. I could do this without even having access to the original data.

Wrap Up

In today’s post we demonstrated how a data scientist can use R with PostgreSQL on their desktop and server-side. The great part was my R code required 0 changes to work in PL/R.

This scenario works well for a company that wants the data scientist working on the statistical algorithm and then, when they finished, expose it in a “safe” way to anyone using the database. By embedding it in PostgreSQL you get four immediate benefits:

The data scientist's R code doesn’t have to be translated to a language that the DBA or application developer understands.
Since the code is running on server with the formula and where the data is stored, your program avoids a whole round trip over the network to do the data analysis.
You get a nice clear separation of concerns; the data scientist gets to build and tune the statistical algorithm and the consumer of the output has no way to alter them.
As long as your functions are ready only, with Postgres read replicas, you get easy horizontal scaling of your algorithm. If you want to give it a try, setting up read replicas is trivial to do on a platform like Crunchy Bridge (which also has PL/R installed).

If you are wondering how an application developer can take advantage of this, tune into the final blog post in the series.

What do you have to say about this? Do you use R in your daily work? Have you ever tried PL/R to do your work? I would love to hear about your experience, share it with us on Twitter. Happy coding and data sciencing!

Using PL/pgSQL to Calculate New Postgres Columns

Steve.Pousty@crunchydata.com (Steve Pousty) — Thu, 10 Dec 2020 04:00:00 EST

In our last blog post on using Postgres for statistics, I covered some of the decisions on how to handle calculated columns in PostgreSQL. I chose to go with adding extra columns to the same table and inserting the calculated values into these new columns. Today’s post is going to cover how to implement this solution using PL/pgSQL.

Why PL/pgSQL?

I bet your first question is going to be: Why did you choose to do this in PL/pgSQL rather than just some quick editor work?

I actually started with handcrafting all the data definition language (DDL) and SQL statements and quickly changed my mind. Just to remind you, here are all the explanatory columns we have in our fire dataset:

Hasfire is the variable we are trying to predict (the occurrence of a fire) and all the other columns are the potential explanatory variables.

Reasons to Replace Copy and Paste with Code

I tried to use code to make the new calculated columns because:

Not only would I have to make 9 new table columns but also 9 different INSERT formula columns. All that is a great opportunity to put a typo in a column name or a formula. It is also a lot of copy and pasting.
Calculating Z-scores is a standard step in many data science workflows. By writing a PL/pgSQL function, I could simply repeat calculating columns in any new analysis by using the function. It would also be available to other data scientists to use as well.
I need to learn more PL/pgSQL and this seemed like an interesting problem. The function was going to have to run some DDL and some SQL.

Design Considerations

As mentioned above, I wanted the function to be generic so I could re-use it in other places. I also decided that it would be best to retain the original column name in the calculated column name. Finally, I wanted to handle all the columns at once, not a function for a single column at a time.

Implementation

With these constraints in mind, I came up with a function that accomplishes all 3 goals. To see the entire function, it's on github, but for the blog post I am going to break it down into sections.

PostgreSQL Function Signature

Let’s start with the function signature:

CREATE OR REPLACE FUNCTION final.initial_center_standardize(schema_name TEXT, table_name TEXT, prefix TEXT, column_names TEXT\[\], pkey TEXT) RETURNS TEXT as

The beginning of the statement is just normal SQL to create a new function. Notice I am actually creating this function inside the schema name “final”. The first two parameters get the schema and table name for the table where we are going to add the columns.

The next parameter is going to be the prefix for the new calculated column name. The next parameter is an array of text values (which is a data type in PostgreSQL). The array contains the name of all the columns for which we want Z-scores. Finally, we need the name of the primary key column for one of the queries used in the function.

I just return a text field because none of this work will return any data values, but I like to show the caller if the call worked. I usually return “done” or “finished” for success and “error” with some text explaining what went wrong.

Just to be clear, for those new to PL/pgSQL programming, I did not come up with this signature at the beginning. Some of the needed parameters only became apparent as I built the rest of the function.

Loop with Iterators in PL/pgSQL

With the signature done, we first declare some variables we will need later in the function. PL/pgSQL requires you to pre-declare any variables you use in a block.

The next step is to create a foreach loop to give us access to each individual element in the array of column names we passed in.

FOREACH col IN ARRAY column_names
    LOOP

Like a normal looping and iterator pattern, this code will extract the first element in the array and give that element the name ‘col’ in the body of the loop. It will continue to iterate through array elements until there are no more elements in the array. This is perfect for our work, each column is going to get the same treatment.

Creating the New Columns

Next our code takes on the task of creating the new column to hold the Z-score. First we create the new column name and storing it in a variable:

new_col_name := prefix || col;

We take the prefix passed in and concatenate it ("||" in SQL) to the original column name. I like using the same prefix before each column because it makes it clearer which columns are the raw columns and which have the Z-score.

Now we do our actual SQL to create the new column in the same table:

EXECUTE FORMAT ('ALTER TABLE %I.%I ADD COLUMN %I NUMERIC', schema_name, table_name, new_col_name);

Normally PL/pgSQL will cache the query plan for a SQL statement. In our case, this statement is going to change each time we run through the loop, making our query dynamic. For dynamic queries you use EXECUTE to tell the PostgreSQL query planner not to cache the query.

To make sure we do SQL safe variable substitution, we use the FORMAT command. The %I in the SQL statement indicates we need this value to be treated like a SQL identifier (i.e. put " around a capitalized table name or escape an ' in a string value). This is the same behavior as the quote_ident command.

Then, by default, the parameters are substituted into the string in the order they appear at the end of the FORMAT command. So if:

schema_name = final
table_name = analysis
new_column_name = z_precip

Then the following statement will be executed:

ALTER TABLE final.analysis ADD COLUMN z_precip NUMERIC

In the specific case of our example we can get away with making every column NUMERIC because Z-scores are always a decimal number regardless of the number type of the input. And with that, we have created our new column in the table.

Populating the Data Values using PL/pgSQL

We have a new column but there is no data in it. Let’s solve that now. Here is the statement I used to calculate the Z-score for the new column (formatted for readability):

EXECUTE FORMAT(
'WITH summary AS ( SELECT AVG(%1$I) AS avg, STDDEV(%1$I) AS stddev FROM %3$I.%4$I), ' ||
'final_select AS (SELECT %5$I, (%1$I - avg)/stddev AS centered FROM %3$I.%4$I CROSS JOIN summary) ' ||
UPDATE %3$I.%4$I SET %2$I = final_select.centered FROM final_select WHERE final_select.%5$I = %3$I.%4$I.%5$I ', col, new_col_name, schema_name, table_name, pkey);

Again we use the EXECUTE FORMAT so the planner doesn’t cache the query and we pass in new variables with each iteration of the loop. The actual SQL has two CTEs to calculate the Z-score and then an UPDATE statement to insert the Z-score into the right table and column.

Because we are reusing parameters in the SQL, we have to use a slightly different syntax for the variable substitution (highlighted in orange). Now we have variables like %1$I rather than just %I. This new notation means take the first parameter from all the parameters passed in to FORMAT. If we had written the query the previous way, we would have passed in the same parameters multiple times in the function call. This numbered reference to parameter format is much cleaner.

The first CTE (summary) calculates the average and standard deviation for the current column in the loop. We take those values and use them in the second CTE (final_select). By using the CROSS JOIN in this query we can use the avg and stddev in the Z-score calculation for each row in the data set. The final query is just updating the new column table with the Z-score where pkey in the final_select matches the pkey of the table.

Closing Out a PL/pgSQL Function

The creation of a new column and populating the values continues for each item in the iteration loop. We terminate the loop with the END LOOP statement. We then return ‘done’ to the SQL call that used the function. Again, I like returning this string because it allows the function user to know if the function actually finished, or it is taking a long time. You could change this to return NULL if you want.

Wrap Up

There you have it, my cool little function. I know it may have looked complicated at first glance, but now that we are finished you can see there are only 2 SQL lines to carry out all our work. Think of how much copy and pasting this saved!

The other big benefit is now I can reuse this function in any database in the future. I can control access to it by using schema permissions. As a matter of fact, I could add this function as part of a template. Then every database created in our hypothetical data science PostgreSQL cluster would have access to this function. I could give data scientists with permission to make this call and quickly calculate Z-scores without even having to know all the PL/pgSQL.

Finally this example could easily be modified to create a new table with new columns for the Z-scores. It could also be quickly modified to use an auxiliary table where the average and standard deviation for each predictive variable is stored. This was one of the possible solutions I explained in the previous post that would probably help overall performance.

I hope you learned something new in this post. Maybe you have already done some PL/pgSQL to alter tables. What kind of use cases did you have? What are something of the interesting twists you uncovered. I would love to hear your thoughts, please sent them to the Crunchy Data twitter account. Happy function writing!

Replacing Lines of Code with 2 Little Regexs in Postgres

Steve.Pousty@crunchydata.com (Steve Pousty) — Wed, 02 Dec 2020 04:00:00 EST

Greetings readers, today we're going to take a semi-break from my “doing data science in SQL” series to cover a really cool use case I just solved with regular expressions (regex) in Postgres. For those of you who have a bad taste in your mouth from earlier run-ins with regexs, this will be more use case focused and I will do my best to explain the search patterns I used.

If you've never heard of regex, there are good resources to learn more about them but I will not be giving a tutorial. I highly recommend learning about them because:

They are available in almost all modern programming languages
They help you solve problems that would be really awkward in code
Most editors also let use regex for find/replace

I have loved regexs ever since I met them. I find them highly logical and easier than writing a bunch of code. I know not all devs agree with me, but to each their own.

Background

I have been doing some work with Island Conservation to help them manage data. I am moving their data off different systems into one centralized PostgreSQL database (with the PostGIS extension as well).

They are doing a study trying to use contraceptives, as opposed to kill trapping, for rodent population management. One of the devices they were using in the field was a camera trap (otherwise known as a game camera).

While the field collector (software on a phone) was giving each camera a GUID for the primary key, the humans would always use the camera name to identify it. Unfortunately, the software was not using controlled vocabularies or value checking for the camera name on the client. All the data values were extracted from the data collector website in GeoJSON format. It was then converted to PostGIS SQL insert statements using ogr2ogr and imported into a table.

The Problem

Now is when things get interesting. Because they were not using a controlled vocabulary, or even formatting requirements on the camera name field, we got all sorts of values there. Some just had “testing” strings in their names and we could ignore them. But others were supposed to follow the format “camXXX” where XXX is 3 digits. The XXX would start with 001 for the first station and then increase accordingly for new stations measured.

Here are just some of the different variations I saw:

Value in Table for camera name	Should be
cam058	This is correct
cam08	cam008
cam59	cam059
CAM011	cam011

The examples above should demonstrate that there are two classes of problems in the data:

Camera names are missing a 0 preceding some of numeric part of the name where the digits are less than 100
The letters in cam are capitalized and should be lowercase

The Solution

Now my first instinct was to say “time to break out some Python and process the original file and fix this”. But my new response kicked in and I said, “Let’s see if we can do this in the database with SQL!”

I start searching the PostgreSQL doc and look what I find! If you search the page you can find regexp_replace which we'll use today (by the way, this doc page should really list all the regexp_* functions). What this commands does is:

Take an input string
Test the regex against the input string
If the regex returns true (the pattern is found), return the replacement string
If it returns false, return the original input string

Ok,, so that fits our requirements quite nicely, time to get writing the regex. Let me be clear again, I am not going to teach you regex today but I will try to walk you through the patterns I used.

Fixing the missing 0s

Alright let’s put one of the incorrect strings here first:

cam08

Here is the SQL I used to fix it

UPDATE camera SET cameraname = regexp_replace(cameraname, 'cam(\d{2})\s\*$',
'cam0\1' )

The SQL parts of this should look like a straightforward UPDATE statement. Let’s look at the regexp_replace function call. The first parameter is the field we want to search, the second parameter is the regex we are using for the test, and the last parameter is the substitution phrase.

Here is the regex by itself:

‘cam(\d{2})\s\*$’

Breaking it down in English:

Look for the string “cam” - capitalization matters in regex unless you tell the command to ignore case.
Now the () means we are going to want to keep whatever is matched inside the parentheses for later
\d means a digit character (0-9)
{2} means we want exactly 2 digits
\s means a space character
- means 0 or more space characters
$ means anchor this pattern at the end of the string (this whole pattern has to occur at the end of the string)

Let me write this as sentences:

At the end of the string look for “cam” followed by exactly 2 digits, save the two digits for later, and there may be trailing spaces.

This exactly matches our records with only 2 numbers rather 3. I also discovered the “hidden” spaces while doing this so I had to put the \s on the end.

I had to anchor to the end so that we wouldn’t match cam001. This anchor is needed because cam00, in cam001, would also match “cam followed by exactly 2 digits followed by 0 or more spaces”.

Now when we look at the third parameter, the replacement string it should look pretty straight forward except the

cam0\1

This says put “cam0” at the beginning of the replacement string. For the \1 we have to go back to step 2 above. That group we put in the () was the first group we captured in the regex. So now \1 contains the content of the first group, which in our case was the 2 digits. If we had another () in the regex that would have been \2.

With this one call we replace all the incorrect camXX values because when the regex matches we get the new string and when it doesn’t we just set the original string back to itself. Boom!

Fixing the Capital Letters

With that other regex under our belt this one should be easier to understand. If you didn’t understand the other one maybe this will help you understand it more. Again, here is our problem string:

CAM011

Again we are going to use regex replace. Here is the regex expression I came up with:

UPDATE camera SET cameraname = regexp_replace(cameraname,
'^CAM(\d{3})\s\*$', 'cam\1')

Here is the regex by itself:

'^CAM(\d{3})\s\*$'

Breaking it down in English:

^ means anchor/start at the beginning of the line
Look for the string “CAM”
() means we are going to want to save the contents inside
\d means means a digit (0-9)
{3} means we want exactly 3 digits
\s means a space
- means 0 or more spaces
$ means this pattern goes to the end of the line

The sentence for this one looks like:

The beginning of the line has to start with “CAM”, followed by exactly 3 digits that we save for later, then 0 or more spaces, and finally this string has to go to the end of the line.

I don’t want to pick up a string like “This is CAM000 test”. To make sure our pattern starts at the beginning of the line, we use ^. We also anchor at the end of the line to ensure the pattern we are searching for is the whole line.

And the replacement parameter is this:

'cam\1'

Since we captured 3 digits in our parentheses, that capture can go right after the ‘cam’ string to make a valid camera name.

Wrap Up

I hope you see both the power and the ease of using regular expressions in the database. Just like programming languages, you can start with simple statements and grow in your expertise over time. There are also some nice web applications to play around and test your regex.

As I said before, you can also use regular expressions in so many other places, like your programming language of choice or your favorite IDE/editor. Do you love regex as much as me? Do you have any cool examples of using them in Postgres? If so shoot us a message on our twitter account. Happy pattern matching!