Uploading Data

This article shows you how to load additional data columns into your TRC sheets. TRC calls additional data columns “semantics”. This article explains the concepts behind semantics and how to upload them in TRC.

Format for semantics

Semantics are CSV files.  This is a universal format and makes it easy to generate from other tools.

They’re either 1 or 2 columns. The first column is always the primary key and used to merge in with the “RecId” of the target sheet.

A semantic can be a 2-column CSV file where the first column is the primary key and second column is data:

semantic_2_col

If a semantic is truly Boolean (“who voted in an election”), you can compress it to a single column of just the ‘active’ RecIds. This can be easy for other tools to generate and reduces upload size.

semantic_1_col

Merging into your sheets

Once you’ve uploaded a semantic to the server, you can merge it into any of your TRC sheets. TRC will merge based on the primary key column (“RecIds”). The order of columns doesn’t matter.

For example, suppose you had a sheet,S1, with 3 rows (R1,R2,R3).  And you had additional data specifying scores for those recIds. You could then add that semantic to the sheet as addititional column, and TRC would merge it in like so:

semantic_merge

You could add the semantic to the sheet and get a new column, “Scores”, with values from the semantics. Since R3 is not in the semantic, it is left blank.
Conceptually, you can think of a semantic merge like Excel’s VLookup function.
You can store “Scores” as a CSV on your Dropbox and refresh it at will. TRC can then pull the updates and refresh your sheet with updated scores. You can merge in multiple semantics and use semantics in filtering and tagging.

Sharing and Security

Semantics are stored and merged on the server.
Generally, each TRC user gets their own directory to store their semantics to avoid conflicting with other users. For example, if you’re user “TU_123” and you upload a semantic named “Scores”, it may get a fully-qualified name (fqn) “users/tu_123/scores”. Then if multiple users both have the same name, the fqn won’t conflict.
You can also share your semantics with others. For example, you may have a microtargeting model for voters, upload that as a semantic, and then share out with a group of candidates using TRC. You might compute the model statewide and have 4 million rows in it; whereas each candidate is only pulling the 80,000 rows that overlap their district. Furthermore, during the final few weeks of the election, you may be recomputing that model daily and pushing out updated models to your field staff daily. TRC sheets can be refreshed daily.

Using the DataUploader plugin

You can upload additional data (semantics) via the “DataUploader” plugin, which you can find on the plugin menu. It looks like this:

datauploader

The “upload” section lets you select a CSV file from Dropbox (see format above) and upload it to TRC. This means your can create the CSV locally using your own tools, write to dropbox via your local file system, and then easily share with TRC. You can update the file and refresh them.
The middle section shows you a) the list of semantics you have access to. If you want to add them to the current sheet, check the box next to the semantic and click “add to sheet”.

Step-By-Step Walkthrough

This is a step-by-step walkthrough for adding a column of data to an existing sheet. This can be used to create additional columns to use in the Filtering tool, or set built-in columns to control targeting, voter matchbacks, and party id.

If you want to create a whole new sheet (such as a new campaign, or new sheet for having different surveys), then contact us.
If you want to just select a subset of rows see Filtering and Targeting.

Part 1: Upload your Semantic

You first need to upload your semantic to the TRC. From here, it can be shared with any sheets you want.

Step 1 – Create a CSV for your data in the proper format.

There are two formats:

  • If you’re just tagging voters, then you need a single column of the RecIds.
  • If you’re appending data values, you’ll need a 2 column file.

See specific examples above.

The file format must be a CSV.

It’s important that your Primary Key matches between this CSV file and the sheet you’re merging with. The Primary Key is the “RecId” column. For example, for voter files, this is often the state voter id. Be sure the first column of your semantic has these same keys so that TRC can properly merge. If your sheet is using state voter IDs, but your semantic’s primary key is county voter ids, then these won’t match and the semantic won’t get merged in properly.

 

Tip: it’s ok if the CSV file has more rows than the intended sheets. TRC will do the merge/joni and take just the rows that are needed.

Step 2 – Copy the CSV to your Dropbox account

You can get a Dropbox account at https://www.dropbox.com

Dropbox is a popular cloud-based file-sharing service and integrates well with your local file system.

Step 3 – Upload your CSV from Dropbox to TRC

Navigate to the Data Uploader tool. You can get here from a sheet’s Plugins menu or from here. 
Use the “Choose from Dropbox” button to select your file from Dropbox. This will take you through Dropbox’s common file chooser dialog.

Upload-dropbox_button

Security Tip: Dropbox’s security will only grant TRC to the file you select here. TRC is still unable to access anything else in your Dropbox.
If you later update the CSV file on Dropbox, TRC can refresh and take the latest copy.

Step 4 – Provide a short name for the new semantic

Provide a short “API” name for the semantic. This is restricted to just alphanumeric characters.

upload-enter_name

Once this is done, you’ll see the semantic listed in the data uploader via its fully qualified name, like “users/tu_12345678/veterans”. The “users/tu_” prefix is what ensures names are globally unique. This becomes very important when you start sharing your semantics with other users.

Congratulations, your new semantic is now securely on the cloud and available to use in TRC sheets!

Part2 : Add your semantic to a sheet

Once your semantic is uploaded to TRC, you can add it to one or more sheets so that you can actually use it.

Navigate to the DataUploader plugin for the sheet you want to add to.

This will let you create a new column in your sheet and bind it to the provided semantic. TRC will do the merge and just extract the appropriate subset of data from your semantic that matches your sheet.

There are two main cases here:
Is this just additional data that you’re using in the filtering tool? In this case, the column name doesn’t matter. See steps 6a.
Else, is this updating one of the built-in columns like PartyId, voter matchback status, etc? In this case, the column name must match the builtin schema. See Step 6b.

Step 6a – Adding columns for filtering

To add additional columns that you can use in the Filtering (https://blog.voter-science.com/2017/06/12/how-to-filter-lists/ ) , Go to the “Existing Semantics” section. This will list all semantics you have access to. Check the box next to the semantic you want to add [1], and then click “Add to sheet” [2]. This will crate a new column name in that sheet based on your semantic name.

upload-add-to-sheet

If a semantic is already present in the sheet, then “Add to sheet?” will show you the column name instead of checkbox [3]. For example, in the example above, the semantic users/tu_1980457901/t5 is already included in the sheet and named column t5.

Step 6b – Updating built-in columns

For builtin columns, the column name must match the schema. The Data Uploader has a “Special Columns” section to help with this.
This lists the various special columns. You can select a semantic from the dropdown [1], and then click “add” [2]. For example, setting the “XTargetPri” column will set the targeted voters.
IF a built-in column is already bound to a semantic, you’ll see that set as the “current value” [3].

upload-set-targets

XVoted and XTargetPri should be a single column semantic that’s just the RecIds to tag.
Party should be a 2 column semantic where the 2nd column is the party value, encoded as 0 through 5.