google-app-engine - Java 中的数据存储导出逻辑
问题描述
值得庆幸的是,谷歌宣布了从云数据存储中导出的逻辑。我想在我的平台上设置时间表导出。但是,它不是 Python,而是 Java。所以我需要使用 cron.xml 和 Java 逻辑来设计这个逻辑。
是否有任何关于 Java 中设计数据存储导出逻辑 (cloud_datastore_admin.py) 的参考?特别是,我需要在 Java 中转换这部分
app = webapp2.WSGIApplication(
[
('/cloud-datastore-export', Export),
], debug=True)
解决方案
您可以按照这些说明为 App Egnine 创建骨架。
一旦你有了骨架,添加这样的东西来处理导出请求:
CloudDatastoreExport.java
package com.google.example.datastore;
import com.google.appengine.api.appidentity.AppIdentityService;
import com.google.appengine.api.appidentity.AppIdentityServiceFactory;
import com.google.apphosting.api.ApiProxy;
import com.google.common.io.CharStreams;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.logging.Logger;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import org.json.JSONArray;
import org.json.JSONObject;
import org.json.JSONTokener;
@WebServlet(name = "CloudDatastoreExport", value = "/cloud-datastore-export")
public class CloudDatastoreExport extends HttpServlet {
private static final Logger log = Logger.getLogger(CloudDatastoreExport.class.getName());
@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
// Verify outputURL parameter
String outputUrlPrefix = request.getParameter("output_url_prefix");
if (outputUrlPrefix == null || !outputUrlPrefix.matches("^gs://.*")) {
response.setStatus(HttpServletResponse.SC_CONFLICT);
response.setContentType("text/plain");
response.getWriter().println("Error: Must provide a valid output_url_prefix.");
} else {
// Get project ID
String projectId = ApiProxy.getCurrentEnvironment().getAppId();
// Remove partition information to get plain app ID
String appId = projectId.replaceFirst("(.*~)", "");
// Get access token
ArrayList<String> scopes = new ArrayList<String>();
scopes.add("https://www.googleapis.com/auth/datastore");
final AppIdentityService appIdentity = AppIdentityServiceFactory.getAppIdentityService();
final AppIdentityService.GetAccessTokenResult accessToken =
appIdentity.getAccessToken(scopes);
// Read export parameters
// If output prefix does not end with slash, add a timestamp
if (!outputUrlPrefix.substring(outputUrlPrefix.length() - 1).contentEquals("/")) {
String timeStamp = new SimpleDateFormat("yyyyMMddHHmmss").format(new Date());
outputUrlPrefix = outputUrlPrefix + "/" + timeStamp + "/";
}
String[] namespaces = request.getParameterValues("namespace_id");
String[] kinds = request.getParameterValues("kind");
// Build export request
JSONObject exportRequest = new JSONObject();
exportRequest.put("output_url_prefix", outputUrlPrefix);
JSONObject entityFilter = new JSONObject();
if (kinds != null) {
JSONArray kindsJSON = new JSONArray(kinds);
entityFilter.put("kinds", kinds);
}
if (namespaces != null) {
JSONArray namespacesJSON = new JSONArray(namespaces);
entityFilter.put("namespaceIds", namespacesJSON);
}
exportRequest.put("entityFilter", entityFilter);
URL url = new URL("https://datastore.googleapis.com/v1/projects/" + appId + ":export");
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setRequestMethod("POST");
connection.addRequestProperty("Content-Type", "application/json");
connection.addRequestProperty("Authorization", "Bearer " + accessToken.getAccessToken());
OutputStreamWriter writer = new OutputStreamWriter(connection.getOutputStream());
exportRequest.write(writer);
writer.close();
if (connection.getResponseCode() == HttpURLConnection.HTTP_OK) {
JSONTokener exportResponseTokens = new JSONTokener(connection.getInputStream());
JSONObject exportResponse = new JSONObject(exportResponseTokens);
response.setContentType("text/plain");
response.getWriter().println("Export started:\n" + exportResponse.toString(4));
} else {
InputStream s = connection.getErrorStream();
InputStreamReader r = new InputStreamReader(s, StandardCharsets.UTF_8);
String errorMessage =
String.format(
"got error (%d) response %s from %s",
connection.getResponseCode(), CharStreams.toString(r), connection.toString());
log.warning(errorMessage);
response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
response.setContentType("text/plain");
response.getWriter().println("Failed to initiate export.");
}
}
}
}
您可以使用cron.yaml
文档中的相同内容:
cron:
- description: "Daily Cloud Datastore Export"
url: /cloud-datastore-export?namespace_id=&output_url_prefix=gs://BUCKET_NAME[/NAMESPACE_PATH]
target: cloud-datastore-admin
schedule: every 24 hours
使用 gcloud 部署 cron 作业:
gcloud app deploy cron.yaml
确保您完成此部分以授予 GAE 导出和存储桶权限,否则您将收到权限被拒绝错误:
https://cloud.google.com/datastore/docs/schedule-export#before_you_begin
推荐阅读
- angularjs - 模块解析失败:/src/app/app.ts 意外令牌 (14:5) 您可能需要适当的加载程序来处理此文件类型
- url-rewriting - Firebase 托管 seo 友好的网址
- javascript - 根据浏览器 userAgent 更改样式
- ubuntu - Hadoop sbin/start-dfs.sh 权限被拒绝
- c# - 在 mvc5 中以默认索引方法输入字符串
- javascript - 仅导致悬停在 svg 中 png 的非透明部分
- django - DJANGO 通过电子邮件重置密码
- javascript - 未捕获的类型错误:document.getElementById(...) 为空
- python - python - 如何使用n个参数和强制验证在python中执行xor的函数?
- shell - 如何在shell脚本中的目录之间迭代