时间戳

首页 > 解决方案 > 使用 boto3 从 AWS Glue 获取表

python - 使用 boto3 从 AWS Glue 获取表

问题描述

我需要从 AWS Glue 爬虫元数据目录中获取表和列名。我使用boto3但不断获得 100 张桌子，即使还有更多。设置NextToken没有帮助。如果可能，请提供帮助。

期望的结果如下：

lst = [table_one.col_one，table_one.col_two，table_two.col_one....table_n.col_n]

def harvest_aws_crawler():
glue = boto3.client('glue', region_name='')
response = glue.get_tables(DatabaseName='', NextToken = '')

#response syntax:
#https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/glue.html#Glue.Client.get_tables
crawler_list_tables = []

for tables in response['TableList']:
    while (response.get('NextToken') is not None):
        crawler_list_tables.append(tables['Name'])
        break
print(len(crawler_list_tables))

harvest_aws_crawler()

更新的代码，仍然需要有表名+列名：

def harvest_aws_crawler():
glue = boto3.client('glue', region_name='')
next_token = ""

#response syntax:
#https://boto3.amazonaws.com/v1/documentation/api/1.9.42/reference/services/glue.html#Glue.Client.get_tables
response = glue.get_tables(DatabaseName='', NextToken = next_token)

tables_from_crawler = []
while True:
    table_list = response['TableList']
    for table_dict in table_list:
        table_name = table_dict['Name']
        
        #append table_name+column_name
        for columns in table_name['StorageDescriptor']['Columns']:
            tables_from_crawler.append(table_name + '.' + columns['Name'])
                
        #tables_from_crawler.append(table_name)
    next_token = response.get('NextToken')
    if next_token is None:
        break
print(tables_from_crawler)

harvest_aws_crawler()

标签： pythonamazon-web-servicespysparkboto3aws-glue

解决方案

您可以使用以下paginator选项尝试以下方法：

def get_tables_for_database(database):
    starting_token = None
    next_page = True
    tables = []
    while next_page:
        paginator = glue_client.get_paginator(operation_name="get_tables")
        response_iterator = paginator.paginate(
            DatabaseName=database,
            PaginationConfig={"PageSize": 100, "StartingToken": starting_token},
        )
        for elem in response_iterator:
            tables += [
                {
                    "name": table["Name"],
                }
                for table in elem["TableList"]
            ]
            try:
                starting_token = elem["NextToken"]
            except:
                next_page = False
    return tables

然后调用该方法列出给定数据库的表：

for table in get_tables_for_database(database):
        print(f"Table: {table['name']}")

如果您想在 Glue 中列出每个数据库的表，您可能必须执行一个额外的 for 循环才能首先检索数据库，然后使用上面的代码片段提取表作为每个数据库的内部循环。

推荐阅读