I'm new to Databricks and when I want to alter my SQL tables by adding primary or FK, I get an absurd error that can't seem to debug it.
%sql
ALTER TABLE samples_abp215dsdp ADD CONSTRAINT PRIMARY (SampleID);
here is the error message:
Error in SQL statement: ParseException:
mismatched input '(' expecting {'CHECK', 'FOREIGN', 'PRIMARY', 'UNIQUE'}(line 1, pos 55)
== SQL ==
ALTER TABLE samples_abp215dsdp ADD CONSTRAINT PRIMARY (SampleID)
------------------------------------------------------^^^
any help will be appreciated.
CodePudding user response:
spark do not have any concept of primary key. As spark is computation engine not database.
There are new features coming in 2022. Follow below list:
- Unity catalog (seems that it will exists next to hive metastore and it will be possible to migrate)
- Control metastore, unity creations etc. from admin console,
- CPU clusters in "create cluster" (similar to SQL endpoints) where you specify how much cpus is needed
- Data lineage
- Delta sharing
- Delta: primary keys
- Delta: foreign keys
- Updated partner connect with many new partners joining
- Notebooks UI: transformations wizards (like select, drop, groupby from droplist etc.)
- New global search
- Integration of job service with git
- Jobs monitoring
- Repair and rerun failed tasks and pass values between tasks
- SQL workflows in SQL view
- create table enhanced via wizard
- Delta live tables in GA
- Serverless DBSQL
- Private links (to VNET and to ExpressRoute) and encryption
- ML: LightGBM, Prophet Models and Text Feature in Auto ML
- ML: Feature store integration with Azure MySQL, Azure SQL and CosmosDB
- ML: Feature store: point in time joins, time-based splits
- ML: Feature store API
- ML: Serverless model serving (enable serving button) with high availability SLA
- ML: one click model deployment into registry
In 2023:
- Delta: identity column
- Delta: dynamic checkpoints and incremental commits to limit read/write operations with ADLS
Refer this link