SQL profile: allow/deny by data type
Is your feature request related to a problem? Please describe.
Column profiling can take extremely long, when Snowflake's VARIANT columns are involved. That may cause errors when the queries time out or otherwise the compute credit consumption rises unnecessarily.
In many cases, metrics like `count(distinct <variant column>)` are also not quite useful.
Describe the solution you'd like
It would be very useful to have a config option to provide allow/deny listing of column datatypes to profile.
For `number(n,m)` formats of different scale/precision, these lists should maybe allow wildcards/regexes as well.
Describe alternatives you've considered
We could just use the existing `profile_pattern.allow/deny` config options, but that would require us to know the names of all VARIANT columns that exist now and in the future.
Subscribe to post
Get notified by email when there are changes.