Alessandro Marrella - bigquery

Avoiding select * in BigQuery

2025-03-19T00:00:00+00:00
A quick workaround to avoid people doing "SELECT *" on your tables (which I consider harmful, see stackoverflow</a>) in BigQuery is the following:</p>
Create a policy tag in the policy tags page</a>. I create it under a new taxonomy "sql_rules", and call it "no_select_star"</li>
Once created, in the policy tag taxonomy page I enable the "enforce access control" option. If you need users to bypass this policy (e.g. needs to run bigquery copy operations) you can add the "fine graded reader" role to the principal associated with the policy tag in the same page.</li>
In the table you want to protect, add a dummy column (I call it "do_not_select_star") and assign it the policy tag</li>
Done! From now on, SELECT *</code> will throw an error.</li> </ol>For more info see Google Cloud docs</a></p>


BigQuery Editions vs On Demand
2024-08-31T00:00:00+00:00
These are some notes on the tradeoffs and best practices between On Demand pricing vs Editions pricing.</p>
</span>Pricing models#</a>
</h2>
BigQuery currently offers two very different pricing models</p>
Editions#</a>
</h3>
With Editions</a> you are charged for "compute time" by slot-hour. A slot is a virtual CPU that BigQuery uses to execute queries.</p>
Within Editions, you can purchase a "committment" for a lower price if you always have stuff running,
with the caveat that you are charged the entire time (it doesn't scale up or down at will, and so for a yearly committment for 100 slots you end up paying for 100 * 365 * 24 slot hours, even if you don't use them).</p>
Outside the committed capacity, you can use an autoscaling reservation, which sets up the minimum (could be 0) and the maximum slots that BigQuery can use.
Slots are scaled up and down based on compute requirements for the queries.</p>
On Demand#</a>
</h3>
With On Demand</a> you are charged for "bytes processed".
The compute capacity that GCP gives you is about 2000 slots, but you are not being charged for it.</p>
Best practices based on the pricing models#</a>
</h2>
Minimize bytes scanned (especially on on demand)#</a>
</h3>

Do not select *, but only select the columns you need.</li>
Partition/Cluster the tables so that you only scan the minimum amount of rows needed (though this increases compute on write, so it's not always helpful with editions, see note on compute used).</li>
Follow the best practices in the bigquery docs (reduce data processed)</a></li>
</ol>
Minimize compute used (especially for editions)#</a>
</h3>

Optimize your joins, see my note about semi hash joins</a>.</li>
Pay attention to when clustering and partitioning are more harmful than helpful.
Take a look at the execution plan of the jobs that produce/update tables with partitioning and clustering.
If the table is large, a lot of slot time is going to be spent on sorting the data to match the clustering.</li>
Follow the best practices in the bigquery docs (optimize query ops)</a></li>
</ol>
Right-size your slots autoscaling (editions only)#</a>
</h3>
Pay particular attention to the jobs where you have a "contention" warning, and dig through the steps in the execution plan.</p>
Check that the "Wait ms" stat and compare it with "Read", "Write" and "Compute".
"Wait" is time that BigQuery spends waiting for slots to be available.</p>
Given that with autoscaling you are charged by the time allocated, and slots are allocated even when they are not used,
wait time still counts towards the cost.</p>
If wait time is too high, this might mean that you need to either:</p>

serialize the queries so that they are no longer in contention for slots</li>
increase the max in the autoscaling reservation to reduce contention</li>
</ul>
Don't increase the autoscaling reservation maximum too much,
as from experience the bigquery autoscaler is very eager to use as many slots as it can
to run the query as quickly as possible, but scaling down takes time (that you are billed for), and the minimum
interval of 1 min adds up quickly if you are using many thousands of slots.</p>
Run queries with the model that's cheapest (if possible)#</a>
</h3>
This sounds obvious, but Google doesn't make it exactly easy. You'll need to:</p>

estimate the price of a query in both modes.</li>
run the query in a dedicated project based on which model you choose (you cannot mix modes within a GCP project)</li>
</ol>
For 1 (estimate the price of a query in both modes), I like to use variations of the following query:</p>
SELECT
</span>job_id, </span>-- the bigquery job id that identifies a run
</span>query, </span>-- the bigquery query text
</span>destination_table</span>.</span>table_id</span>, </span>-- the name of the table (if the query writes to a table)
</span>start_time,
</span>SUM</span>(total_slot_ms) / (</span>1000 </span>* </span>60 </span>* </span>60</span>) </span>* </span>0</span>.</span>06 </span>as editions_cost </span>-- unit cost (here $0.06/slot_hour) could be different based on the edition chosen and gcp discounts
</span>SUM</span>(total_bytes_billed) / POW(</span>1024</span>, </span>4</span>) </span>* </span>6</span>.</span>25 </span>as on_demand_cost </span>-- unit cost (here $6.25/tb) could be different based on the edition chosen and gcp discounts
</span>FROM </span>`</span>{your_project}</span>`.`</span>{your_region}</span>`.</span>INFORMATION_SCHEMA</span>.</span>JOBS_BY_PROJECT </span>-- or JOBS_BY_ORGANIZATION to see the whole company, but then you'll need to remove the query field
</span>GROUP BY</span> ALL
</span></code></pre>

Note that the prices here are estimated, because google bills by slots "assigned" not slots used
so sometimes slot price is always slightly higher than the estimate (it takes time to scale up and down,
and google bills a minimum of 1 minute even if a query runs for 3 seconds).</p>
</blockquote>
For 2 (run the query in a dedicated project), you'll need to set up two separate GCP projects,
one with the reservation (or the reservation assigned from another project) and the other without.</p>


BigQuery performance best practice: use semi joins when possible
2024-06-02T00:00:00+00:00
SQL is an amazing language, it lets you declaratively say what you want, and the engine figures out for you the best way to return it to you. Or should I say, it figures out the best way to return it to you given the information it has and the capabilities of the engine itself</em>.</p>
In this post, we’ll discuss a performance optimization technique for BigQuery (also other advanced enough Enteprise Data Warehouses and databases support SEMI JOINS, but I'll focus on BigQuery since it's the one I use the most these days): using semi joins. </p>
</span>

A SEMI JOIN returns rows from the first table (left table) where one or more matches are found in the second table (right table), but it does not return rows from the second table. This can significantly improve query performance in certain scenarios.”</p>
</blockquote>
Let's look at an example query using their public datasets, the NYC taxi dataset, which contains a log of taxi trips in NYC.</p>
Suppose that we want to know the distinct pickup_location_id values where both yellow and green taxis picked up clients in 2022.</p>
One way to express this query might be the following:</p>
SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>  </span>join </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span>  on </span>yellow</span>.</span>pickup_location_id </span>= </span>green</span>.</span>pickup_location_id
</span></code></pre>
If we look at the performance, on my project at the time of running this query's performance is the following (results might change a bit based on your google cloud project, slot availability, time of day, etc.):</p>

Total elapsed time</strong>: 4min 45sec</li>
Slot time consumed</strong>: 3h40min (A slot</a> in BigQuery is a unit of computational capacity required to execute SQL queries)</li>
</ul>
If we look at the execution graph, we see the following


</p>
The join is definitely the step that takes the longest! If we click the join step in the graph, BigQuery really helpfully shows us more information:</p>


The UI shows that the join produced a lot more rows than it consumed, due to how the join was applied. You can also see that it uses a INNER HASH JOIN</code>. </p>
In theory, this is a very efficient kind of join, as it builds a hash table with the join keys of one table (the smaller one), and then probes the other table (the larger one) to find the row keys that have a match in the hash table.</p>
The problem in this case is not much in how the join happens, but in what it produces. As you can peek from the screenshot above, the number of rows produced is 83 million! This is due to the many-to-many relationship that we have in this join, where a pickup_location_id</code> can happen multiple times in either table.</p>
In a sense, BigQuery here does way more than we need to, as it would be enough to find one match in the "green" table to consider the row in yellow valid. In other words, we need a SEMI JOIN</code>. </p>

A SEMI JOIN</code> returns rows from the first table (left table) where one or more matches are found in the second table (right table), but it does not return rows from the second table. </p>
</blockquote>
How do we rewrite the previous query to make BigQuery use a semi join?</p>
For this specific query we have (at least) three options, which all make use of the SEMI HASH JOIN</code> in the query plan:</p>
Option 1: use EXISTS</code>#</a>
</h3>
SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>  </span>WHERE </span>EXISTS (
</span>    </span>SELECT </span>1
</span>    </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span>    </span>WHERE </span>green</span>.</span>pickup_location_id </span>= </span>yellow</span>.</span>pickup_location_id
</span>  )
</span></code></pre>
Option 2: use IN</code>#</a>
</h3>
SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>  </span>WHERE </span>yellow</span>.</span>pickup_location_id </span>IN (
</span>    </span>SELECT DISTINCT </span>green</span>.</span>pickup_location_id
</span>    </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span>  )
</span></code></pre>
Option 3: use INTERSECT DISTINCT</code>#</a>
</h3>
SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>INTERSECT DISTINCT 
</span>SELECT DISTINCT </span>green</span>.</span>pickup_location_id
</span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span></code></pre>
Comparison#</a>
</h3>
The three options all return the same result of the original query, and they all produce a query plan with a semi hash join,
with a much better performance:</p>

Total elapsed time</strong>: 1sec</li>
Slot time consumed</strong>: 40sec</li>
</ul>
While they generate a similarly shaped plan, and in this case produce the same result, they are not the same from a logical point of view.</p>
EXISTS#</a>
</h4>
Option 1 (EXISTS</code>) is the most flexible, because it lets you write multiple predicates in the WHERE clause. So you can for example  write:</p>
SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>  </span>WHERE </span>EXISTS (
</span>    </span>SELECT </span>1
</span>    </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span>    </span>WHERE </span>green</span>.</span>pickup_location_id </span>= </span>yellow</span>.</span>pickup_location_id
</span>    </span>-- add another predicate
</span>    AND </span>green</span>.</span>dropoff_location_id </span>= </span>yellow</span>.</span>dropoff_location_id
</span>  )
</span></code></pre>
This would still do a SEMI HASH JOIN</code>, but now we also filter the rows so that the result returns locations where both the pickup and dropoff was the same.</p>
From a performance point of view, even if this is fast, this still scans 83,869,625</code> rows in the join phase.</p>
If we want to reduce the number of rows scanned, in this case we can do it with a WITH statement, like this:</p>
WITH</span> green AS (
</span>  </span>SELECT DISTINCT</span> pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>`
</span>)
</span>SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>  </span>WHERE </span>EXISTS (
</span>    </span>SELECT </span>1
</span>    </span>FROM</span> green
</span>    </span>WHERE </span>green</span>.</span>pickup_location_id </span>= </span>yellow</span>.</span>pickup_location_id
</span>  )
</span></code></pre>
This query usually performs a bit faster (for me it's around 900ms), uses slightly less slot seconds (for me about 30s), and scans less rows in the join phase (now 36,272,535</code>). 
The main difference is that we do a DISTINCT</code> before joining.</p>
To reduce the join to a minimum, we can also do one more step and do a distinct on yellow too, and we get</p>
WITH</span> green AS (
</span>  </span>SELECT DISTINCT</span> pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>`
</span>),
</span>yellow AS (
</span>  </span>SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>)
</span>SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM</span> yellow
</span>  </span>WHERE </span>EXISTS (
</span>    </span>SELECT </span>1
</span>    </span>FROM</span> green
</span>    </span>WHERE </span>green</span>.</span>pickup_location_id </span>= </span>yellow</span>.</span>pickup_location_id
</span>  )
</span></code></pre>
Now the performance is even faster (for me around 800ms), uses even less slot seconds (about 20s), and more deterministically we can say that it only scans 16,813</code>in the join phase.</p>
IN#</a>
</h4>
With IN</code> we are a bit more constrained (unless we do ugly string concatenation things), as we can really only compare one element per statement.</p>
So to express a query where we want pickup_location_id</code> and dropoff_location_id</code> to be the same you'd have to write:</p>
SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>  </span>WHERE </span>yellow</span>.</span>pickup_location_id </span>IN (
</span>    </span>SELECT DISTINCT </span>green</span>.</span>pickup_location_id
</span>    </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span>  )
</span>  AND </span>yellow</span>.</span>dropoff_location_id </span>IN (
</span>    </span>SELECT DISTINCT </span>green</span>.</span>dropoff_location_id
</span>    </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span>  )
</span></code></pre>
Since this uses two statements, in the JOIN</code> step in the query it does two joins!</p>


The IN</code> is more perfomant than the original EXISTS</code> query written above, and is more comparable to the EXISTS</code> query where we use WITH green AS...</code> to do the select distinct. Also in the IN</code> case we scan 36,272,535</code>rows (like in the first improved exists). </p>
We can reach a comparable performance to the second improved exists (the one with WITH green AS..., yellow AS...</code>) if we do </p>
WITH</span> yellow AS (
</span>  </span>SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2022</span>` yellow
</span>)
</span>SELECT DISTINCT </span>yellow</span>.</span>pickup_location_id
</span>FROM</span> yellow
</span>WHERE </span>yellow</span>.</span>pickup_location_id </span>IN (
</span>  </span>SELECT DISTINCT </span>green</span>.</span>dropoff_location_id
</span>  </span>FROM </span>`</span>bigquery-public-data.new_york_taxi_trips.tlc_green_trips_2022</span>` green
</span>)
</span></code></pre>
Also here, we scan 16,819</code> rows in the JOIN.</p>
There is one more caveat with IN</code>, as sometimes IN</code> and EXISTS</code> don't always produce the same result! See NOT IN and NOT EXISTS don't always produce the same result</a></p>
INTERSECT DISTINCT#</a>
</h4>
I'll admit it, I crafted the query so that intersect distinct would make the cut as well, as I particularly like it syntax wise.</p>
INTERSECT DISTINCT</code> in our case produces exactly the same result as IN</code> and EXISTS</code> but has a different limitation: the columns selected need to be the same in both tables AND the result needs to be distinct (there is no such thing as an INTERSECT ALL</code>).</p>
From a performance profile, this immediately produces the most efficient result, scanning only  16,819</code> rows in the JOIN (which again, is a SEMI HASH JOIN</code>).</p>
Conclusion#</a>
</h3>
In conclusion, we saw that SEMI HASH JOIN</code> can be a powerful optimization, especially when dealing with many-to-many relationships, and can be done in several ways. Each method has its strengths and limitations, but in the end the main optimization is the impact on the query plan of moving from a INNER</code> to a SEMI</code> join.</p>


NOT IN and NOT EXISTS don't always produce the same result
2024-06-01T00:00:00+00:00
IN</code> and EXISTS</code> often produce the same result, but when negated and dealing with NULL</code> values, they behave differently.</p>
</span>
Let's see an example, assume we have these two tables:</p>
orders</strong></p>
order_id</th> customer_id</th> amount</th></tr></thead>

1</td> 100</td> 50.00</td></tr>
2</td> 101</td> 75.00</td></tr>
3</td> 102</td> 30.00</td></tr>
4</td> NULL</td> 20.00</td></tr>
</tbody></table>
customers</strong></p>
customer_id</th> name</th></tr></thead>

100</td> Bob</td></tr>
101</td> Alice</td></tr>
102</td> Martha</td></tr>
NULL</td> John</td></tr>
</tbody></table>
Ignore the fact that a well formed customers table would need to have the id always specified and ideally as a constraint. This is just an example.</em></p>
Now, let’s say we want to find orders where the customer_id is not present in the customers table.</p>
Using NOT EXISTS</code> we would write something like:</p>
SELECT </span>*
</span>FROM</span> orders o
</span>WHERE</span> NOT EXISTS (</span>SELECT </span>1 </span>FROM</span> customers c </span>WHERE </span>c</span>.</span>customer_id </span>= </span>o</span>.</span>customer_id</span>);
</span></code></pre>
The output, would probably be what we expect:</p>
order_id</th> customer_id</th> amount</th></tr></thead>

4</td> NULL</td> 20.00</td></tr>
</tbody></table>
Using NOT IN</code> we would write something like:</p>
SELECT </span>*
</span>FROM</span> orders
</span>WHERE</span> customer_id NOT IN (</span>SELECT</span> customer_id </span>FROM</span> customers);
</span></code></pre>
The output here will be empty!</p>
When we use NOT IN</code>, SQL checks each value in the orders table against the list of values returned by the subquery. If any value in the subquery result is NULL</code>, the entire NOT IN</code> comparison will result in NULL</code> for each row in the orders table. This is because any comparison with NULL</code> yields NULL</code> and NOT IN</code> needs all comparisons</strong> to be TRUE</code> for a row to be included in the result.</p>
This is not a problem with NOT EXISTS</code>, because the  NOT EXISTS</code> clause checks for the non-existence of rows that meet the criteria specified in the subquery. It does not perform direct comparisons with NULL</code> in the same way NOT IN</code> does. Instead, it simply checks if there are any rows that match the condition. If no such rows exist, the condition is true.</p>
Alessandro Marrella - bigquery

Avoiding select * in BigQuery

BigQuery Editions vs On Demand

On Demand#</a> </h3> With On Demand</a> you are charged for "bytes processed". The compute capacity that GCP gives you is about 2000 slots, but you are not being charged for it.</p>

BigQuery performance best practice: use semi joins when possible

Comparison#</a> </h3> The three options all return the same result of the original query, and they all produce a query plan with a semi hash join, with a much better performance:</p> Total elapsed time</strong>: 1sec</li>

On Demand
On Demand</a> you are charged for "bytes processed". The compute capacity that GCP gives you is about 2000 slots, but you are not being charged for it.</p>
