java - 如何最好地在 hadoop 上模拟用户帐户

Question

我有一个将文件从 Unix 复制到 hdfs 的 Java 程序。它运行良好，但是我正在寻找在运行和复制文件时模拟不同的帐户。

输入：除了输入文件和目标hdfs目录路径，另一个输入应该是包含帐户，keytab目录，域的属性文件

请让我知道前进的最佳方式。

我目前正在探索使用 shell 首先发出 kinit 命令然后运行 ​​jar

我还在阅读有关 Jaas 以及如何在 Java 本身中做到这一点 - 来自 - https://henning.kropponline.de/2016/02/14/a-secure-hdfs-client-example/

需要输入和可用选项的任何参考。

我的复制文件的Java程序如下：

public class FileCopy implements Runnable {

@Option(names = {"-i","--input"}, required=true, description="file name to copy to hadoop")
String input;

@Option(names = {"-o","--output"}, required=true, description="hdfs directory path to be copied into")
String output;


public void run() {


    Properties hadoop_properties = new Properties();
    HdfsFileDeploy hdfsFileDeploy = new HdfsFileDeploy();

    try {
        hadoop_properties.load(FileCopy.class.getClassLoader().getResourceAsStream("hadoop.properties"));
    } catch (IOException e) {
        e.printStackTrace();
    }


    FileSystem fs = hdfsFileDeploy.configureFilesystem(hadoop_properties.getProperty("coreSitePath"),hadoop_properties.getProperty("hdfsSitePath"));


    String status = hdfsFileDeploy.writeToHDFS(fs,input,output);

    if (status == "SUCCESS") {
            System.out.println("completed copying");
    } else {
        System.out.println("copying error");
    }

    hdfsFileDeploy.closeFileSystem(fs);

}

public static void main(String[] args) throws IOException {


    CommandLine.run(new FileCopy(), args);

}

}

公共类 HdfsFileDeploy {

   public FileSystem configureFilesystem(String coreSitePath, String hdfsSitePath) {

        FileSystem fileSystem = null;

        try {

            Configuration conf = new Configuration();
            Path hdfsCoreSitePath = new Path(coreSitePath);
            Path hdfsHDFSSitePath = new Path(hdfsSitePath);
            conf.addResource(hdfsCoreSitePath);
            conf.addResource(hdfsHDFSSitePath);


            fileSystem = FileSystem.get(conf);
            System.out.println(fileSystem);
            return fileSystem;

        } catch (Exception ex) {

            ex.printStackTrace();
            return fileSystem;
        }
    }


   public void closeFileSystem(FileSystem fileSystem) {

        try {
            fileSystem.close();
        } catch (Exception ex) {
                System.out.println("Unable to close Hadoop filesystem : " + ex);

        }
   }

   // 

    public String writeToHDFS(FileSystem fileSystem, String sourcePath, String destinationPath) {

        String failure = "FAILURE";
        String success = "SUCCESS";
        Boolean doNotDelSrc = false;
        Boolean overwrite = true;

        try {
            Path inputPath = new Path(sourcePath);
            Path outputPath = new Path(destinationPath);

            if(!fileSystem.exists(outputPath)) {
                System.out.println("Output path " + outputPath + " does not exist. Creating outputPath directory now..");
                if (fileSystem.mkdirs(outputPath)) {
                    System.out.println("Output path " + outputPath + " created...");
                }
            }

            System.out.println("about to copy from " + inputPath + " to " + outputPath);
            fileSystem.copyFromLocalFile(doNotDelSrc, overwrite, inputPath, outputPath);
            return success;

        } catch (IOException ex) {

            System.out.println("Some exception occurred while writing file to hdfs");
            ex.printStackTrace();
            return failure;
        }
    }

}

Input1：输入文件 Input2：目标 hdfs 目录 参考输入：文件（比如 yaml）包含帐户、域、keytab 路径。

jar 应该模拟输入文件并将其复制到目标 hdfs 目录。

标签： javahadoopkerberos