TMG Data Library

TMG data library has the functionalities to interact with Google Cloud services allowing to develop more reliable and standard data pipelines.

Quick Start

Installation

Install this library in a virtualenv using pip.

Supported Python Versions

Python >= 3.5

Mac/Linux

pip install virtualenv
virtualenv <your-env>
source <your-env>/bin/activate
<your-env>/bin/pip install tmg-data

Example Usage

Transform from MySQL to BigQuery

from tmg.data import transfer

transfer_client = transfer.Client(project='your-project-id')

transfer_client.bq_to_mysql(
    connection_string='root:password@host:port/your-database',
    bq_table='your-project-id.your-dataset.your-table',
    mysql_table='your-database.your-table'
)

Changelog

0.0.1

  • TMG data library is born

0.0.2

  • Adding logs client

0.0.3

  • Remove basic config from logger

0.0.4

  • Set allow_quoted_newlines to true for big query job loads

0.0.5

  • Modified ftp module to use paramiko and updated test cases

0.0.6

  • Using Jinja templating for writing query in BigQuery

  • Fix running query from file

0.0.7

  • Added compression functionality during GBQ table exporting process to GCS on bq_to_gs()

0.0.8

  • Added Amazon aws s3 storage functionalities - download, upload

  • Added s3 to gs and gs to s3 functionality in transfer module

0.0.9

  • Adding Salesforce functionality

0.1.0

  • remove unused panda library

0.1.1

  • cleanup library dependencies

0.1.2

  • Add BigQuery partitioning functionality

  • Fix querying from external tables

  • Fix unittest

  • Fix Salesforce initialisation issue

0.1.3

  • Add write preference argument for running bq queries

  • Make write preference required argument for most of the transfer functions

  • Fix Parallel running

0.1.4

  • Add name argument to the Parallel running add function

  • Add ability to transform Salesforce data on fields.

  • Fix bq upload to not use autodetect when schema is specified

0.1.5

  • Add ability to transform Salesforce data on fields for transfer module.

0.1.6

  • Fix memory issue to load Salesforce big table by using generator

  • Add logging for running BigQuery queries and showing total byte processed.

0.1.7

  • Added port for MySQL connection string

0.1.8

  • Add better logging for running queries on BigQuery.

0.1.9

  • Add max_bad_records parameter to gs_to_bq().

  • Updated gs_uri to gs_uris and added sequence[str] as a valid data type in gs_to_bq().

0.1.10

  • Fix exception handling in Salesforce download table functionality.

0.1.11

  • update ftp_to_bq function to accept schema for the destination table.

0.1.12

  • Add max_bad_records parameter to BigQuery upload table.

0.1.13

  • Add multi factor authentication for FTP module

0.1.14

  • Fix raising error in Runner module.

0.1.16

  • Fix multi factor authentication FTP module

0.1.17

  • Add bq upload to salesforce

  • Add salesforce upload function

0.1.18

  • Bug Fix on markupsafe library version bump

0.1.19

  • Fix Salesforce update_salesforce to not exit when any update fails