com.socrata.api

Class SodaImporter

java.lang.Object

com.socrata.api.SodaWorkflow

com.socrata.api.SodaDdl

com.socrata.api.SodaImporter

public class SodaImporter
extends SodaDdl

This class contains all the apis for using the full file import/update apis. The update and append APIs in this class require the dataset is in a working copy. Since, creating and publishing working copies can be expensive, when operating on large datsets or doing frequent updates you should use the Soda2Producer class. Soda2Producer does not require creating working copies, however, when doing very large changes or replacing a dataset working copies can be useful. Look at http://dev.socrata.com/publishers/workflow for information about the workflow process.

Field Summary

Fields

Modifier and Type	Field and Description
static String	SCAN_BASE_PATH

Fields inherited from class com.socrata.api.SodaDdl

ASSET_BASE_PATH, COLUMNS_PATH, SEARCH_BASE_PATH

Fields inherited from class com.socrata.api.SodaWorkflow

API_BASE_PATH, GEO_BASE_PATH, geocodingUri, httpLowLevel, mapper, TICKET_CHECK, VIEWS_BASE_PATH, viewUri

Constructor Summary

Constructors

Constructor and Description
SodaImporter(HttpLowLevel httpLowLevel) Constructor.

Method Summary

Methods

Modifier and Type	Method and Description
DatasetInfo	append(String datasetId, File file, int skip, String[] translation) This appends the contents of a file to a dataset on Socrata.
DatasetInfo	createViewFromCsv(String name, String description, File file) Creates a dataset from a CSV, using all the default column types.
DatasetInfo	createViewFromCsv(String name, String description, File file, String rowIdentifierColumnName) Creates a dataset from a CSV, using all the default column types.
String[]	generateTranslation(Blueprint blueprint) Creates a straight translation with no transforms for a given bluprint.
NonDataFileDataset	importNonDataFile(String name, String description, File file) Imports a file that is NOT going to be used to create a datset, but is instead available for downloading directly.
DatasetInfo	importScanResults(Blueprint blueprint, String[] translation, File file, ScanResults scanResults) Imports the results of scanning a file.
DatasetInfo	importScanResults(String name, String description, File file, ScanResults scanResults) Imports the results of scanning a file.
DatasetInfo	importScanResults(String name, String description, File file, ScanResults scanResults, String rowIdentifierColumnName) Imports the results of scanning a file.
static SodaImporter	newImporter(String url, String userName, String password, String token) Create a new SodaImporter object, using the supplied credentials for authentication.
DatasetInfo	replace(String datasetId, File file, int skip, String[] translation) This replaces the contents of a file to a dataset on Socrata.
ScanResults	scan(File file) Scans a file, then sends it up to the Socrata service to be analyzed and have things like column types guessed.
protected DatasetInfo	sendScanResults(String basePostBody, String fileId, String[] translation, File file)
protected DatasetInfo	updateFromScanResults(String datasetId, String method, int skip, String fileId, String[] translation, File file)

Methods inherited from class com.socrata.api.SodaDdl

addAsset, addColumn, alterColumn, createDataset, deleteDataset, getAsset, loadDatasetInfo, newDdl, removeColumn, searchViews, updateDatasetInfo

Methods inherited from class com.socrata.api.SodaWorkflow

createWorkingCopy, findPendingGeocodingResults, getHttpLowLevel, makePrivate, makePublic, newWorkflow, publish, waitForPendingGeocoding

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SCAN_BASE_PATH

public static final String SCAN_BASE_PATH

See Also:

Constant Field Values

Constructor Detail

SodaImporter

public SodaImporter(HttpLowLevel httpLowLevel)

Constructor.

Parameters:

httpLowLevel - the HttpLowLevel this uses to contact the server

Method Detail

newImporter

public static final SodaImporter newImporter(String url,
                       String userName,
                       String password,
                       String token)

Create a new SodaImporter object, using the supplied credentials for authentication.

Parameters:

url - the base URL for the SODA2 domain to access.

userName - user name to log in as

password - password to log in with

token - the App Token to use for authorization and usage tracking. If this is null, no value will be sent.

Returns:

fully configured SodaImporter

createViewFromCsv

public DatasetInfo createViewFromCsv(String name,
                            String description,
                            File file,
                            @Nullable
                            String rowIdentifierColumnName)
                              throws InterruptedException,
                                     SodaError,
                                     IOException

Creates a dataset from a CSV, using all the default column types. This will also assume the CSV has a single header row at the top.

Parameters:

name - name of the dataset to create

description - description of the new dataset

file - the file to upload

rowIdentifierColumnName - row identifie

Returns:

return the view that was just created.

Throws:

InterruptedException

SodaError

IOException

createViewFromCsv

public DatasetInfo createViewFromCsv(String name,
                            String description,
                            File file)
                              throws InterruptedException,
                                     SodaError,
                                     IOException

Creates a dataset from a CSV, using all the default column types. This will also assume the CSV has a single header row at the top.

Parameters:

name - name of the dataset to create

description - description of the new dataset

file - the file to upload

Returns:

return the view that was just created.

Throws:

InterruptedException

SodaError

IOException

scan

public ScanResults scan(File file)
                 throws SodaError,
                        InterruptedException

Scans a file, then sends it up to the Socrata service to be analyzed and have things like column types guessed.

Parameters:

file - File to upload

Returns:

the results of the scan.

Throws:

SodaError

InterruptedException

importScanResults

public DatasetInfo importScanResults(String name,
                            String description,
                            File file,
                            ScanResults scanResults)
                              throws SodaError,
                                     InterruptedException,
                                     IOException

Imports the results of scanning a file. This will build a default blueprint from it, assuming the first rows are column names.

Parameters:

name - name of the dataset to create

description - description of the datset

file - file that was scanned

scanResults - results of the scan

Returns:

The default View object for the dataset that was just created.

Throws:

SodaError

InterruptedException

IOException

importScanResults

public DatasetInfo importScanResults(String name,
                            String description,
                            File file,
                            ScanResults scanResults,
                            @Nullable
                            String rowIdentifierColumnName)
                              throws SodaError,
                                     InterruptedException,
                                     IOException

Imports the results of scanning a file. This will build a default blueprint from it, assuming the first rows are column names.

Parameters:

name - name of the dataset to create

description - description of the datset

file - file that was scanned

scanResults - results of the scan

Returns:

The default View object for the dataset that was just created.

Throws:

SodaError

InterruptedException

IOException

importScanResults

public DatasetInfo importScanResults(Blueprint blueprint,
                            String[] translation,
                            File file,
                            ScanResults scanResults)
                              throws SodaError,
                                     InterruptedException,
                                     IOException

Imports the results of scanning a file. This method does not assume anything about the CSV, but instead has the caller provide the blueprint and the translation for any schema defintion or data transforms.

Parameters:

blueprint -

translation -

file - file that was scanned

scanResults - results of the scan

Returns:

The default View object for the dataset that was just created.

Throws:

SodaError

InterruptedException

IOException

append

public DatasetInfo append(String datasetId,
                 File file,
                 int skip,
                 String[] translation)
                   throws SodaError,
                          InterruptedException,
                          IOException

This appends the contents of a file to a dataset on Socrata. This operation requires the dataset is in a working copy, so unless you are doing large updates or your dataset is small, using the UPSERT functionality in Soda2Producer may give you better results. If you are doing frequent updates, the apis in Soda2Producer may give better results (since they don't require working copies)

Parameters:

datasetId - id of the dataset to append to

file - file with the data in it

skip - number of rows in the data to skip (normally for skipping headers)

translation - an optional translation array for translating from values in the file and values in the dataset.

Returns:

The info of the dataset after the append operation.

Throws:

SodaError

InterruptedException

IOException

replace

public DatasetInfo replace(String datasetId,
                  File file,
                  int skip,
                  String[] translation)
                    throws SodaError,
                           InterruptedException,
                           IOException

This replaces the contents of a file to a dataset on Socrata. This operation requires the dataset is in a working copy, which is an expensive operation. If your dataset is large, you may want to figure out how to figure out which rows to update, rather than doing a full replace for updates. If you are doing frequent updates, the apis in Soda2Producer may give better results (since they don't require working copies)

Parameters:

datasetId - id of the dataset to append to

file - file with the data in it

skip - number of rows in the data to skip (normally for skipping headers)

translation - an optional translation array for translating from values in the file and values in the dataset.

Returns:

The info of the dataset after the append operation.

Throws:

SodaError

InterruptedException

IOException

updateFromScanResults

protected DatasetInfo updateFromScanResults(String datasetId,
                                String method,
                                int skip,
                                String fileId,
                                String[] translation,
                                File file)
                                     throws SodaError,
                                            InterruptedException,
                                            IOException

Throws:

SodaError

InterruptedException

IOException

sendScanResults

protected DatasetInfo sendScanResults(String basePostBody,
                          String fileId,
                          String[] translation,
                          File file)
                               throws SodaError,
                                      InterruptedException,
                                      IOException

Throws:

SodaError

InterruptedException

IOException

importNonDataFile

public NonDataFileDataset importNonDataFile(String name,
                                   String description,
                                   File file)
                                     throws SodaError,
                                            InterruptedException

Imports a file that is NOT going to be used to create a datset, but is instead available for downloading directly.

Parameters:

name - name of the file

description - description of the file

file - the file to upload

Returns:

The NonDataFileDataset object that was saved to Socrata

Throws:

SodaError

InterruptedException

generateTranslation

public String[] generateTranslation(Blueprint blueprint)

Creates a straight translation with no transforms for a given bluprint.

Parameters:

blueprint - blueprint to build the translation from

Returns:

the array of mappings to map each field to itself. This will create a translation that will do nothing.