首页 > 技术文章 > hive 自动加载分区 --动态分区

students 2020-04-16 21:51 原文


#!/bin/bash

date1="20080101"

date2="20080401"

enddate="20200501"

while [ "$date1" -le "$enddate" ];

do

yesterday=`date -d "$date1" +%Y-%m-%d`

today=`date -d "$date2" +%Y-%m-%d`

echo "${yesterday}" "${today}"


hive -e ' use bd; set hive.exec.max.dynamic.partitions.pernode=100; set hive.exec.dynamici.partition=true; set hive.exec.dynamic.partition.mode=nonstrict; insert overwrite table ods_xxx partition(sale_day) select * from (select *,substr(sale_time,0,10) from tmp_ods_xxx where substr(sale_time,0,10) > "2020-01-25")t ;'


date1=`date -d "$date1 +3month" +%Y%m%d`

date2=`date -d "$date2 +3month" +%Y%m%d`

done

 

对于动态分区,hive最多一次插入100个分区

推荐阅读