What to pass in expression field of Glue API while doing get

so I am trying to get partitions from my Glue table. But I need only a certain set of partitions that pass the condition that is applied to the partition's value.

I did try the other way of getting all the partitions and then filtering them out, but it is making a lot of API calls.

I checked the boto3 documentation but I don't think that I could apply a filter expression on the partition's value. Nor could I find any example of a filter expression being applied.

The value of partition is as follows [1, '202210051', 1, 123]

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.get_partitions

CodePudding user response：

The get_partitions will only pull partitions that matches the expression. I have partitions of below format

dt=2022-09-07/partition_1=08
dt=2022-09-08/partition_1=07
dt=2022-09-12/partition_1=06
dt=2022-09-16/partition_1=05

If I want to filter for only partition then the expression will be of below format:

expression="dt = '2022-09-07' AND partition_1 = 08"

response = client.get_partitions(
    
    DatabaseName='test_glue',
    TableName='testahana_ahana_test_test_ql_8vuvs70w9h',
    Expression=expression,
    MaxResults=123
)

Similarly depending on your requirement you can pass the conditional expression to get_partitions