Tag Archives: google-bigquery

Copying data from BigQuery to Postgres (Cloud SQL)

Posted on July 10, 2023 by Siddharth Jain

Google provides a database migration service but at time of this writing it does not support copying data from BigQuery to Postgres (Cloud SQL). You can DIY in a couple of ways. Here I describe a way to do it … Continue reading →

Posted in Computers, programming, Software | Tagged , | Leave a comment

Get BigQuery Table Schema

Posted on June 20, 2023 by Siddharth Jain

This post describes how to get the schema of a table in BigQuery. Its actually quite straightforward (I mean you can use standard JDBC methods to get it) if one uses the BigQuery JBDC connector like below: BigQueryJdbcConnection is a … Continue reading →

Posted in Computers, programming, Software | Tagged , | Leave a comment

Measuring performance of BigQuery vs. Clickhouse

Posted on December 10, 2022 by Siddharth Jain

This post summarizes results of a performance test to compare BigQuery vs. Clickhouse – two high-performance analytics databases. Later on I performed same tests on DuckDB but as we shall see it cannot compete with CH or BQ so main … Continue reading →

Posted in Computers, programming, Software | Tagged , | Leave a comment

BigQuery Cheat Sheet

Posted on August 10, 2022 by Siddharth Jain

2. List all tables within a dataset 3. List schema of a table aliter: to get as json: 4. Create a dataset: 5. Get job details. Two ways of doing this. Method1: use bq show command: Method2: use REST endpoint. … Continue reading →

Posted in Computers, programming, Software | Tagged | Leave a comment

Measuring performance of BigQuery Java vs Python connector

Posted on August 1, 2022 by Siddharth Jain

While experimenting with Apache Beam, I noticed there is a big difference in the performance of Java vs. Python SDK when reading from BigQuery. See this. So I thought it would be good to measure the performance of the Java … Continue reading →

Posted in Computers, programming, Software | Tagged | Leave a comment

How to process data in Google BigQuery?

Posted on July 4, 2022 by Siddharth Jain

In this post we discuss some of the ways the data in BigQuery can be processed. First, for those unfamiliar with it, what is BigQuery? BigQuery is Google’s OLAP (Online Analytics Processing) database. In my previous company, we used Hive … Continue reading →

Posted in Computers, programming, Software | Tagged , | Leave a comment

Query Google BiqQuery from a Scala console app

Posted on June 19, 2022 by Siddharth Jain

Objective How can we query Google BigQuery from a Scala console app? Step 1 – Create project on Google CLoud, enable billing, Install Google Cloud SDK, initialize and authenticate Refer to online documentation for how to do all this. You … Continue reading →

Posted in Computers, programming, Software | Tagged , | Leave a comment