select column if not exists return as null

I have two possibilities to get data from two different tables: Table A and Table B.

The table B doesn't have one column(ColumnD), so I wanna to keep the full schema

I tried using Python(Spark):

#input argument
table_argument = 'B'

spark.sql(f'''
SELECT 
 ColumnA,
 ColumnB,
 ColumnC,
 CASE
   WHEN '{table_argument}' = 'B' THEN NULL
 ELSE ColumnD 
 END AS ColumnD
FROM {table_argument}''')

I got always the ERROR that is not possible to get the ColumnD. The When condition didn't work.

Is there any possibility to do this?

Thanks

CodePudding user response：

You can't reference a nonexistent column, even in a WHEN clause that's not executed. You need to use a Python condition to construct that part of the query.

spark.sql(f'''
SELECT 
 ColumnA,
 ColumnB,
 ColumnC,
 {'NULL' if table_argument == 'B' else 'ColumnD'} AS ColumnD
FROM {table_argument}''')