{"id":23542,"date":"2025-11-13T18:35:41","date_gmt":"2025-11-13T18:35:41","guid":{"rendered":"https:\/\/pokecon.jp\/job\/?p=23542"},"modified":"2025-11-13T18:35:41","modified_gmt":"2025-11-13T18:35:41","slug":"repro%e3%81%a7%e3%81%aeapache-iceberg%e3%81%ae%e6%8a%80%e8%a1%93%e6%a4%9c%e8%a8%bc%e7%b5%90%e6%9e%9c-repro-tech-blog","status":"publish","type":"post","link":"https:\/\/pokecon.jp\/job\/23542\/","title":{"rendered":"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c &#8211; Repro Tech Blog"},"content":{"rendered":"\n<\/p>\n<div>\n<p>Repro\u3067\u30c1\u30fc\u30d5\u30a2\u30fc\u30ad\u30c6\u30af\u30c8\u3068\u3057\u3066\u4ed5\u4e8b\u3092\u3057\u3066\u3044\u308b\u6a4b\u7acb(joker1007)\u3067\u3059\u3002<\/p>\n<p>\u4eca\u56de\u3001\u672c\u756a\u74b0\u5883\u3078\u306e\u5c0e\u5165\u3092\u8996\u91ce\u306b\u5165\u308c\u3066<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/Apache\">Apache<\/a> Iceberg\u3092\u5b9f\u969b\u306b\u696d\u52d9\u30c7\u30fc\u30bf\u3067\u691c\u8a3c\u3057\u307e\u3057\u305f\u3002<\/p>\n<p>\u3053\u306e\u8a18\u4e8b\u3067\u306f\u305d\u306e\u691c\u8a3c\u5185\u5bb9\u3068\u7d50\u679c\u306b\u3064\u3044\u3066\u5171\u6709\u3057\u305f\u3044\u3068\u601d\u3044\u307e\u3059\u3002<\/p>\n<p>Iceberg\u306fOpen Table Format\u3068\u547c\u3070\u308c\u3066\u3044\u308b\u30c7\u30fc\u30bf\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u306e\u4e00\u3064\u3067\u3001<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%A6%A5%C9\">\u30af\u30e9\u30a6\u30c9<\/a>\u74b0\u5883\u3067\u52b9\u7387\u3088\u304f\u30c7\u30fc\u30bf\u5206\u6790\u3092\u884c\u3046\u305f\u3081\u306e\u30c7\u30fc\u30bf\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u3068\u3057\u3066\u958b\u767a\u304c\u9032\u3093\u3067\u3044\u307e\u3059\u3002<\/p>\n<p>Iceberg\u306e\u69d8\u306aOpen Table Format\u306f\u5358\u4e00\u306e\u30d5\u30a1\u30a4\u30eb\u306b\u3088\u308b\u30c7\u30fc\u30bf\u69cb\u9020\u3067\u306f\u306a\u304f\u3001\u8907\u6570\u306e\u30d5\u30a1\u30a4\u30eb\u3068\u305d\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%E1%A5%BF%A5%C7%A1%BC%A5%BF\">\u30e1\u30bf\u30c7\u30fc\u30bf<\/a>\u306e\u7ba1\u7406\u6a5f\u69cb\u304c\u5408\u308f\u3055\u3063\u305f\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u306b\u306a\u3063\u3066\u304a\u308a\u3001\u5168\u4f53\u306e\u30d5\u30a1\u30a4\u30eb\u69cb\u9020\u304c\u5408\u308f\u3055\u3063\u3066\u4e00\u3064\u306e\u30c6\u30fc\u30d6\u30eb\u3068\u3057\u3066\u6271\u3048\u308b\u3068\u3044\u3046\u306e\u304c\u7279\u5fb4\u3067\u3059\u3002<\/p>\n<p>\u4ee5\u524d\u3001Repro\u3067\u691c\u8a3c\u3057\u305f<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/Apache\">Apache<\/a> Hudi\u3082Open Table Format\u306e\u4e00\u3064\u3067\u3059\u304c\u3001Iceberg\u306fHudi\u3068\u6bd4\u3079\u3066\u30b7\u30f3\u30d7\u30eb\u306a\u4ed5\u7d44\u307f\u306b\u306a\u3063\u3066\u3044\u308b\u5370\u8c61\u3067\u3059\u3002<\/p>\n<p>Open Table Format\u7fa4\u304c\u65e7\u6765\u306e\u30c7\u30fc\u30bf\u69cb\u9020\u306b\u6bd4\u3079\u3066\u512a\u308c\u3066\u3044\u308b\u70b9\u306e\u4e00\u3064\u3068\u3057\u3066\u3001\u5206\u6790\u7528\u9014\u306b\u5411\u3044\u305f\u5927\u898f\u6a21\u30c7\u30fc\u30bf\u3092\u524d\u63d0\u306b\u3057\u3066\u3044\u306a\u304c\u3089\u30ec\u30b3\u30fc\u30c9\u5358\u4f4d\u306eUPDATE\u3084DELETE\u306b\u5bfe\u5fdc\u3057\u3066\u3044\u308b\u3068\u3044\u3046\u70b9\u3067\u3059\u3002\u5927\u91cf\u30c7\u30fc\u30bf\u51e6\u7406\u306b\u5fc5\u8981\u306a\u306e\u306f\u30a4\u30d9\u30f3\u30c8\u30ed\u30b0\u306e\u69d8\u306bINSERT\u3060\u3051\u306b\u5bfe\u5fdc\u3067\u304d\u3066\u3044\u308c\u3070\u826f\u3044\u3082\u306e\u3082\u3042\u308c\u3070\u3001\u66f4\u65b0\u304c\u5fc5\u8981\u306a\u30c7\u30fc\u30bf\u3082\u3042\u308a\u307e\u3059\u3002\u4eca\u56de\u306e\u691c\u8a3c\u3067\u5229\u7528\u3057\u305f\u30e6\u30fc\u30b6\u30fc\u5358\u4f4d\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%E1%A5%BF%A5%C7%A1%BC%A5%BF\">\u30e1\u30bf\u30c7\u30fc\u30bf<\/a>\u306a\u3069\u3082\u305d\u3046\u3044\u3063\u305f\u6027\u8cea\u306e\u30c7\u30fc\u30bf\u3067\u3059\u3002<br \/>\n\u3053\u3046\u3044\u3063\u305f\u30c7\u30fc\u30bf\u3092\u6271\u3046\u305f\u3081\u306b\u306f\u3001\u65e7\u6765\u306f\u30c7\u30fc\u30bf\u306e\u6295\u5165\u306e\u5f8c\u306b\u91cd\u8907\u3092\u6392\u9664\u3059\u308b\u30af\u30a8\u30ea\u3092\u66f8\u304d\u3064\u3064\u3001\u5b9a\u671f\u7684\u306b\u30c6\u30fc\u30d6\u30eb\u3092\u6d17\u3044\u66ff\u3048\u3059\u308b\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u3092\u5b9f\u65bd\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3057\u305f\u3002\u305d\u308c\u3092\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u81ea\u4f53\u306e\u4ed5\u7d44\u307f\u3068\u66f8\u304d\u8fbc\u307f\u306e\u305f\u3081\u306e\u57fa\u76e4\u3067\u30b5\u30dd\u30fc\u30c8\u3057\u3066\u304f\u308c\u308b\u3068\u3044\u3046\u306e\u304cOpen Table Format\u306e\u30e1\u30ea\u30c3\u30c8\u3067\u3059\u3002<\/p>\n<p>\u3061\u306a\u307f\u306b\u3001\u3053\u306e\u8a18\u4e8b\u3067\u306fIceberg\u304c\u3069\u3046\u3044\u3063\u305f\u30c7\u30fc\u30bf\u69cb\u9020\u306a\u306e\u304b\u3068\u3044\u3046\u8a73\u7d30\u306f\u89e3\u8aac\u3057\u307e\u305b\u3093\u306e\u3067\u3001\u516c\u5f0f\u30c9\u30ad\u30e5\u30e1\u30f3\u30c8\u3068\u3001\u5185\u90e8\u69cb\u9020\u306b\u3064\u3044\u3066\u5206\u304b\u308a\u6613\u304f\u89e3\u8aac\u3057\u3066\u304f\u308c\u3066\u3044\u308b\u8a18\u4e8b\u3078\u306e\u30ea\u30f3\u30af\u3092\u53c2\u8003\u8cc7\u6599\u3068\u3057\u3066\u63b2\u8f09\u3057\u3066\u304a\u304d\u307e\u3059\u3002<\/p>\n<h2 id=\"\u53c2\u8003\u8cc7\u6599\">\u53c2\u8003\u8cc7\u6599<\/h2>\n<p>Repro\u3067\u306f\u5927\u4f53200\u5104\u4ef6\u3050\u3089\u3044\u306e\u30ec\u30b3\u30fc\u30c9\u3092\u6301\u3064\u30c6\u30fc\u30d6\u30eb\u304b\u30891000\u4e07\u4ef6\u524d\u5f8c\u3092\u53d6\u5f97\u3059\u308b\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u3068\u3001\u666e\u901a\u306bWeb\u30ea\u30af<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%A8%A5%B9\">\u30a8\u30b9<\/a>\u30c8\u30671\u4ef6\u5358\u4f4d\u3067\u53d6\u5f97\u3059\u308b\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u304c\u6df7\u5728\u3057\u3066\u3044\u307e\u3059\u3002<br \/>\n\u305d\u3057\u3066\u3001\u305d\u306e\u30c7\u30fc\u30bf\u306f\u51fa\u6765\u308b\u9650\u308a\u77ed\u3044\u30ea\u30fc\u30c9\u30bf\u30a4\u30e0\u3067\u5229\u7528\u53ef\u80fd\u306b\u306a\u3063\u3066\u3044\u308b\u3053\u3068\u304c\u671b\u307e\u3057\u3044\u3082\u306e\u3067\u3059\u3002(5\u5206\u4ee5\u5185\u3050\u3089\u3044\u306a\u3089\u8a31\u5bb9\u53ef\u80fd)<\/p>\n<p>\u3053\u306e\u76ee\u7684\u306e\u305f\u3081\u306b\u3001Repro\u3067\u306fCassandra\u3092\u30d0\u30c3\u30af\u30a8\u30f3\u30c9\u3068\u3057\u3001\u30c6\u30fc\u30d6\u30eb\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u30ad\u30fc\u53ca\u3073<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF%A5%EA%A5%F3%A5%B0\">\u30af\u30e9\u30b9\u30bf\u30ea\u30f3\u30b0<\/a>\u30ad\u30fc\u3092\u5de5\u592b\u3057\u3066bucketing\u306a\u3069\u306e\u30c6\u30af\u30cb\u30c3\u30af\u3092\u6d3b\u7528\u3059\u308b\u3053\u3068\u3067\u306a\u3093\u3068\u304b\u5bfe\u5fdc\u3067\u304d\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u5927\u898f\u6a21\u306a\u8aad\u307f\u8fbc\u307f\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u3067\u306fTrino\u3092\u5229\u7528\u3057\u3001\u305d\u306e\u4ed6\u306e\u30ea\u30af<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%A8%A5%B9\">\u30a8\u30b9<\/a>\u30c8\u3067\u306f\u666e\u901a\u306bCassandra Client\u304b\u3089\u8aad\u307f\u53d6\u308a\u3092\u884c\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u3057\u304b\u3057\u3001Cassandra\u306f\u30d0\u30eb\u30af\u30ed\u30fc\u30c9\u306b\u5411\u3044\u305f\u69cb\u9020\u3068\u306f\u8a00\u3044\u96e3\u304f\u3001\u73fe\u72b6Trino\u306e\u4e26\u5217\u8aad\u307f\u8fbc\u307f\u3068\u30c6\u30fc\u30d6\u30eb\u306ebucketing\u306b\u3088\u308a\u7121\u7406\u77e2\u7406\u306a\u3093\u3068\u304b\u3057\u3066\u3044\u308b\u3068\u3044\u3046\u5074\u9762\u304c\u3042\u308a\u307e\u3059\u3002<br \/>\n\u73fe\u72b6\u5bfe\u5fdc\u306f\u51fa\u6765\u3066\u3044\u308b\u304c\u7121\u99c4\u304c\u591a\u3044\u3057\u3001\u5c06\u6765\u7684\u306a\u30b9\u30b1\u30fc\u30e9\u30d3\u30ea\u30c6\u30a3\u306b\u4e00\u5b9a\u306e\u4e0d\u5b89\u304c\u3042\u308b\u72b6\u614b\u306a\u306e\u3067\u3001\u3088\u308a\u826f\u3044\u57fa\u76e4\u306b\u7f6e\u304d\u63db\u3048\u3066\u3044\u304d\u305f\u3044\u3068\u3044\u3046\u306e\u304c\u5b9f\u60c5\u3067\u3059\u3002<\/p>\n<p>\u3057\u304b\u3057\u3001\u958b\u767a\u30e1\u30f3\u30d0\u30fc\u304c\u904b\u7528\u3067\u304d\u308b\u57fa\u76e4\u306b\u3082\u9650\u308a\u304c\u3042\u308b\u306e\u3067\u3001\u3067\u304d\u308b\u9650\u308a\u904b\u7528\u8ca0\u8377\u3092\u4e0a\u3052\u306a\u3044\u3053\u3068\u3082\u5927\u4e8b\u306b\u3057\u305f\u3044\u3068\u3053\u308d\u3067\u306f\u3042\u308a\u307e\u3059\u3002<\/p>\n<p>\u305d\u3046\u3044\u3046\u8a33\u3067\u3001\u60c5\u5831\u3092\u96c6\u3081\u3066\u3044\u305f\u306e\u3067\u3059\u304c\u3001\u4eca\u56de\u4e0a\u8a18\u306e\u8981\u6c42\u3092\u6e80\u305f\u305b\u305d\u3046\u306a\u9078\u629e\u80a2\u3068\u3057\u3066iceberg\u306b\u6ce8\u76ee\u3057\u691c\u8a3c\u3092\u884c\u3044\u307e\u3057\u305f\u3002<\/p>\n<p>\u691c\u8a3c\u5bfe\u8c61\u3068\u3057\u3066\u91cd\u8996\u3057\u305f\u306e\u306f\u4ee5\u4e0b\u306e\u8981\u7d20\u3067\u3059\u3002<\/p>\n<ol>\n<li>\u5f0a\u793e\u306e\u30c7\u30fc\u30bf\u51e6\u7406\u57fa\u76e4\u306e\u4e2d\u6838\u306b\u3042\u308bKafka\u304b\u3089\u306e\u66f8\u304d\u8fbc\u307f\u304c\u7c21\u5358\u3067\u3042\u308b\u3053\u3068<\/li>\n<li>Upsert\u51e6\u7406\u304c\u53ef\u80fd\u3067\u3042\u308b\u3053\u3068<\/li>\n<li>\u30c7\u30fc\u30bf\u81ea\u4f53\u3092S3\u306b\u914d\u7f6e\u53ef\u80fd\u3067\u3042\u308b\u3053\u3068<\/li>\n<li>\u65e2\u5b58\u57fa\u76e4\u3068\u306e\u7d44\u307f\u5408\u308f\u305b\u3092\u542b\u3081\u3066\u30015\u5206\u4ee5\u5185\u306b\u306f\u30c7\u30fc\u30bf\u304c\u30af\u30a8\u30ea\u53ef\u80fd\u306b\u306a\u308b\u898b\u8fbc\u307f\u304c\u3042\u308b\u3053\u3068<\/li>\n<\/ol>\n<p>\u5b9f\u969b\u306e\u3068\u3053\u308d\u3084\u3063\u3066\u307f\u306a\u3044\u3068\u5206\u304b\u3089\u3093\u306a\u3068\u3044\u3046\u306e\u304c4\u3064\u76ee\u3067\u3001Repro\u306e\u30c7\u30fc\u30bf\u91cf\u53ca\u3073\u66f8\u304d\u8fbc\u307f\u306e\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u306b\u5bfe\u3057\u3066\u3001iceberg\u306e\u66f8\u304d\u8fbc\u307f\u3068commit\u306b\u304b\u304b\u308b\u30ce\u30fc\u30c9\u306e\u30ea\u30bd\u30fc\u30b9\u30b3\u30b9\u30c8\u304c\u3069\u306e\u7a0b\u5ea6\u306b\u306a\u308b\u306e\u304b\u3001commit\u3068compaction\u304c\u3069\u308c\u3050\u3089\u3044\u306e\u5b9f\u884c\u983b\u5ea6\u306a\u3089\u30d0\u30e9\u30f3\u30b9\u304c\u53d6\u308c\u308b\u306e\u304b\u3002compaction\u306b\u3069\u308c\u3050\u3089\u3044\u306e\u8ca0\u8377\u304c\u304b\u304b\u308b\u306e\u304b\u3001\u3053\u306e\u8fba\u308a\u3092\u77e5\u308b\u305f\u3081\u306b\u5b9f\u969b\u306e\u30c7\u30fc\u30bf\u3067\u691c\u8a3c\u304c\u5fc5\u8981\u3067\u3057\u305f\u3002<\/p>\n<p>&#8212; \u4ee5\u4e0b\u3001\u793e\u5185\u5411\u3051\u306e\u8a55\u4fa1\u5831\u544a\u3092\u624b\u76f4\u3057\u3057\u305f\u3082\u306e\u306a\u306e\u3067\u6587\u4f53\u304c\u5909\u308f\u308a\u307e\u3059 &#8212;<\/p>\n<p>\u73fe\u884c\u306ecassandra\u3078\u306e\u66f8\u304d\u8fbc\u307f\u3068\u540c\u69d8\u306bKafka\u304b\u3089\u30ec\u30b3\u30fc\u30c9\u3092\u53d7\u3051\u53d6\u3063\u3066iceberg\u30c6\u30fc\u30d6\u30eb\u306e\u66f8\u304d\u8fbc\u307f\u3092\u884c\u3046\u3002<br \/>\n\u30b3\u30df\u30c3\u30c8\u53ca\u3073compaction\u51e6\u7406\u304cRepro\u306e\u6a5f\u80fd\u30fb\u904b\u7528\u9762\u3067\u73fe\u5b9f\u7684\u306a\u983b\u5ea6\u306e\u5b9f\u884c\u3067\u51e6\u7406\u304c\u53ef\u80fd\u304b\u3069\u3046\u304b\u3092\u691c\u8a3c\u3057\u3001production\u3067\u5fc5\u8981\u306b\u306a\u308b\u30ce\u30fc\u30c9\u6570\u3092\u898b\u7a4d\u308b\u3002<\/p>\n<p>\u307e\u305f\u3001trino\u3067\u30af\u30a8\u30ea\u3092\u3059\u308b\u6642\u306e\u7c21\u5358\u306a\u30af\u30a8\u30ea\u30d1\u30bf\u30fc\u30f3\u306e\u5b9f\u9a13\u30fb\u691c\u8a0e\u3092\u884c\u3044\u6319\u52d5\u3092\u78ba\u8a8d\u3059\u308b\u3002<\/p>\n<p>\u30ec\u30b3\u30fc\u30c9\u306e\u5185\u5bb9\u306f\u30a8\u30f3\u30c9\u30e6\u30fc\u30b6\u30fc\u306b\u95a2\u3059\u308b<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%E1%A5%BF%A5%C7%A1%BC%A5%BF\">\u30e1\u30bf\u30c7\u30fc\u30bf<\/a>\u3092\u4fdd\u6301\u3059\u308buser_id+\u30ad\u30fc\u30d0\u30ea\u30e5\u30fc\u3068\u3044\u3046\u30b7\u30f3\u30d7\u30eb\u306a\u69cb\u9020\u306e\u30ec\u30b3\u30fc\u30c9\u3067\u3042\u308a\u3001Update\u3082\u983b\u7e41\u306b\u767a\u751f\u3059\u308b\u3002<\/p>\n<p>\u540c\u4e00\u306e\u5185\u5bb9\u3067\u3042\u308c\u3070\u51e6\u7406\u3092\u30b9\u30ad\u30c3\u30d7\u3059\u308b\u4ed5\u7d44\u307f\u306f\u65e2\u306b\u524d\u6bb5\u306b\u69cb\u7bc9\u4e2d\u3067\u3042\u308b\u305f\u3081\u3001\u672c\u5f53\u306b\u66f8\u304d\u8fbc\u307f\u3092\u5fc5\u8981\u3068\u3059\u308b<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%B9%A5%EB%A1%BC%A5%D7%A5%C3%A5%C8\">\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8<\/a>\u306f\u305d\u3053\u307e\u3067\u9ad8\u304f\u306a\u3044\u3002\u5927\u4f53\u79d2\u95936000\u4ef6\u7a0b\u5ea6\u304c\u30ab\u30d0\u30fc\u3067\u304d\u308c\u3070\u73fe\u72b6\u306f\u5bfe\u5fdc\u53ef\u80fd\u3002<\/p>\n<h2 id=\"\u66f8\u304d\u8fbc\u307f\u65b9\u5f0f\">\u66f8\u304d\u8fbc\u307f\u65b9\u5f0f<\/h2>\n<p>EMR-7.10\u3092\u5229\u7528\u3057\u3066Flink\u3067S3\u306b\u30c7\u30fc\u30bf\u3092\u66f8\u304d\u8fbc\u3080\u3002<\/p>\n<h2 id=\"\u30c6\u30fc\u30d6\u30eb\u5b9a\u7fa9\u3068\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\">\u30c6\u30fc\u30d6\u30eb\u5b9a\u7fa9\u3068\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9<\/h2>\n<p>\u30c6\u30fc\u30d6\u30eb\u306e\u7ba1\u7406\u306b\u306f<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/AWS\">AWS<\/a> Glue Data Catalog\u3092\u5229\u7528\u3057\u3001spark-<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/sql\">sql<\/a>\u3092\u5229\u7528\u3057\u305f<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/DDL\">DDL<\/a>\u3067\u69cb\u7bc9\u3059\u308b\u3002<\/p>\n<p>Glue Data Catalog\u3092\u5229\u7528\u3059\u308b\u306e\u306f\u3001Catalog\u7ba1\u7406\u306e\u904b\u7528\u30b3\u30b9\u30c8\u3092\u524a\u6e1b\u3059\u308b\u305f\u3081\u3068\u3001Glue\u304c\u6301\u3064Iceberg\u306e\u30c6\u30fc\u30d6\u30eb\u6700\u9069\u5316\u306e\u6a5f\u80fd\u3092\u5229\u7528\u3057\u305f\u3044\u305f\u3081\u3002<\/p>\n<p>\u307e\u305f\u3001iceberg\u306f\u7279\u6027\u3068\u3057\u3066\u3044\u304f\u3064\u304b\u306e\u5b9a\u671f\u7684\u306a\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u30bf\u30b9\u30af\u304c\u5fc5\u8981\u306b\u306a\u308b\u3002\u7279\u306bFlink\u304b\u3089\u77ed\u3044\u30b9\u30d1\u30f3\u3067\u66f8\u304d\u8fbc\u307f\u3092\u884c\u3046\u3068\u5c0f\u3055\u3044\u30b5\u30a4\u30ba\u306eparquet\u30d5\u30a1\u30a4\u30eb\u3068Update\u306b\u5bfe\u5fdc\u3059\u308b\u305f\u3081\u306e\u524a\u9664\u30d5\u30a1\u30a4\u30eb\u304c\u5927\u91cf\u306b\u767a\u751f\u3059\u308b\u305f\u3081\u3001\u5b9a\u671f\u7684\u306bcompaction\u3092\u884c\u308f\u306a\u3044\u3068\u8aad\u307f\u53d6\u308a\u52b9\u7387\u304c\u5927\u304d\u304f\u52a3\u5316\u3059\u308b\u3002<\/p>\n<p>\u305d\u306e\u305f\u3081\u5b9f\u884c\u3057\u306a\u3051\u308c\u3070\u306a\u3089\u306a\u3044\u306e\u304c\u4ee5\u4e0b\u306e3\u3064\u3067\u3042\u308b\u3002\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u30bf\u30b9\u30af\u306f\u4e00\u822c\u7684\u306b\u306fspark\u304b\u3089\u5b9f\u884c\u3059\u308b\u3002spark-<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/sql\">sql<\/a>\u3092\u5229\u7528\u3059\u308b\u306e\u304c\u6700\u3082\u7c21\u5358\u3067\u3042\u308b\u3002<\/p>\n<ul>\n<li>expire_snapshot: iceberg\u306f\u30b3\u30df\u30c3\u30c8\u3054\u3068\u306bsnapshot (manifest)\u3092\u4f5c\u6210\u3059\u308b\u306e\u3067\u3001\u7121\u9650\u306b\u84c4\u7a4d\u3057\u306a\u3044\u69d8\u306b\u4e00\u5b9a\u4ee5\u4e0a\u53e4\u3044snapshot\u3092\u524a\u9664\u3059\u308b\u3002<\/li>\n<li><a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/rewrite\">rewrite<\/a>_data_files (compaction): merge-on-read\u306e\u305f\u3081\u306b\u751f\u6210\u3055\u308c\u305f\u7d30\u5207\u308c\u306eparquet\u30d5\u30a1\u30a4\u30eb\u3084delete\u60c5\u5831\u30d5\u30a1\u30a4\u30eb\u3092\u7d50\u5408\u3059\u308b\u3002<\/li>\n<li>delete_orphan_files: \u4e0a\u8a18\u306e\u30bf\u30b9\u30af\u306b\u3088\u3063\u3066\u4e0d\u8981\u306b\u306a\u3063\u305f\u5b9f\u4f53\u30d5\u30a1\u30a4\u30eb\u3092\u524a\u9664\u3059\u308b\u3002<\/li>\n<\/ul>\n<h2 id=\"\u524a\u9664\u30d5\u30a1\u30a4\u30eb\u306b\u3064\u3044\u3066\u88dc\u8db3\">\u524a\u9664\u30d5\u30a1\u30a4\u30eb\u306b\u3064\u3044\u3066\u88dc\u8db3<\/h2>\n<p>\u5192\u982d\u3067\u8ff0\u3079\u305f\u69d8\u306bIceberg\u306fAnalytics\u7528\u9014\u306b\u5411\u3044\u305f\u30c7\u30fc\u30bf\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u3067\u3042\u308a\u3064\u3064Update(Delete)\u3092\u30b5\u30dd\u30fc\u30c8\u3057\u3066\u3044\u308b\u306e\u304c\u7279\u5fb4\u3067\u3059\u3002\u3053\u3046\u3044\u3063\u305f\u30c7\u30fc\u30bf\u30d5\u30a9\u30fc\u30de\u30c3\u30c8\u3067Update\u3092\u30b5\u30dd\u30fc\u30c8\u3059\u308b\u306b\u306f\u4ee3\u8868\u7684\u306a\u4e8c\u3064\u306e\u65b9\u6cd5\u304c\u3042\u308a\u307e\u3059\u3002<\/p>\n<ol>\n<li>Copy on Write: Update\u8981\u6c42\u304c\u3042\u3063\u305f\u6642\u306b\u8a72\u5f53\u30ec\u30b3\u30fc\u30c9\u3092\u542b\u3080\u5217\u6307\u5411\u30c7\u30fc\u30bf\u30d5\u30a9\u30fc\u30de\u30c3\u30c8(\u57fa\u672c\u7684\u306b\u306fParquet)\u306e\u30d5\u30a1\u30a4\u30eb\u3092\u4e38\u3054\u3068\u7f6e\u304d\u63db\u3048\u308b<\/li>\n<li>Merge on Read : Update\u8981\u6c42\u306b\u8a72\u5f53\u3059\u308b\u30ec\u30b3\u30fc\u30c9\u306e\u524a\u9664\u304c\u3042\u3063\u305f\u3053\u3068\u3092\u8a18\u9332\u3057\u3001\u65b0\u3057\u3044\u5024\u3092\u6301\u3064\u30ec\u30b3\u30fc\u30c9\u3092\u6301\u3063\u305f\u65b0\u898f\u30d5\u30a1\u30a4\u30eb\u3068\u524a\u9664\u8a18\u9332\u30d5\u30a1\u30a4\u30eb\u3092\u4f5c\u6210\u3001\u8aad\u307f\u53d6\u308a\u6642\u306b\u305d\u308c\u3089\u3092\u81ea\u52d5\u7684\u306b\u30de\u30fc\u30b8\u3059\u308b<\/li>\n<\/ol>\n<p>Merge on Read\u306f\u983b\u7e41\u306aUpdate\u304c\u767a\u751f\u3059\u308b\u5834\u5408\u306b\u5411\u3044\u305f\u5f62\u5f0f\u3067\u3001\u66f8\u304d\u8fbc\u307f\u6642\u306b\u306f\u65b0\u3057\u304f\u5c0f\u3055\u3044\u30d5\u30a1\u30a4\u30eb\u3092\u4f5c\u308b\u3060\u3051\u3067\u6e08\u3080\u306e\u3067Flink\u306a\u3069\u304b\u3089\u30b9\u30c8\u30ea\u30fc\u30e0\u3067\u77ed\u3044\u30ea\u30fc\u30c9\u30bf\u30a4\u30e0\u3067\u30c7\u30fc\u30bf\u3092\u66f8\u304d\u8fbc\u3080\u969b\u306b\u76f8\u6027\u304c\u826f\u3044\u65b9\u5f0f\u3067\u3059\u3002\u4e00\u65b9\u3067\u5c0f\u3055\u3044\u30d5\u30a1\u30a4\u30eb\u304c\u5927\u91cf\u306b\u767a\u751f\u3059\u308b\u3053\u3068\u3001\u305d\u3057\u3066\u524a\u9664\u5185\u5bb9\u3092\u8a18\u9332\u3057\u305f\u30d5\u30a1\u30a4\u30eb\u304c\u5fc5\u8981\u306b\u306a\u308b\u305f\u3081\u66f4\u306b\u30d5\u30a1\u30a4\u30eb\u6570\u304c\u5897\u3048\u3066\u8aad\u307f\u53d6\u308a\u6642\u306e\u7121\u99c4\u3082\u5927\u304d\u304f\u306a\u308b\u306e\u304c\u30c7\u30e1\u30ea\u30c3\u30c8\u3067\u3059\u3002<\/p>\n<p>Iceberg\u3067\u306f\u3001\u3053\u3046\u3044\u3063\u305f\u5c0f\u3055\u3044\u30b5\u30a4\u30ba\u306e\u30c7\u30fc\u30bf\u30d5\u30a1\u30a4\u30eb\u3068\u524a\u9664\u30d5\u30a1\u30a4\u30eb\u306e\u5897\u5927\u3092\u4e00\u5b9a\u306e\u7bc4\u56f2\u3067\u53ce\u3081\u308b\u305f\u3081\u3001\u5b9a\u671f\u7684\u306bcompaction\u3068\u547c\u3070\u308c\u308b\u51e6\u7406\u3092\u5b9f\u884c\u3057\u3066Parquet\u30d5\u30a1\u30a4\u30eb\u3092\u30de\u30fc\u30b8\u3057\u51e6\u7406\u52b9\u7387\u3092\u7dad\u6301\u3057\u306a\u3051\u308c\u3070\u306a\u308a\u307e\u305b\u3093\u3002<\/p>\n<h2 id=\"\u30c7\u30fc\u30bf\u898f\u6a21\">\u30c7\u30fc\u30bf\u898f\u6a21<\/h2>\n<p>\u5168\u4f53\u306e\u30c7\u30fc\u30bf\u91cf: 2TB\u7a0b<br \/>\n\u30ec\u30b3\u30fc\u30c9\u6570: 200\u5104\u4ef6\u8d85<br \/>\n\u66f4\u65b0\u983b\u5ea6: 5000\u4ef6 \/ sec<\/p>\n<p>bucketing\u306e\u7d50\u679c\u3001\u30d0\u30e9\u3064\u304d\u306f\u3042\u308b\u304c\u5927\u4f531<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u304c\u5e73\u57471GB\u3050\u3089\u3044\u306b\u306a\u308b\u69d8\u306b\u3057\u3066\u691c\u8a3c\u3002(<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D0%A5%B1%A5%C3%A5%C8\">\u30d0\u30b1\u30c3\u30c8<\/a>\u30b5\u30a4\u30ba\u3001\u30d5\u30a1\u30a4\u30eb\u30b5\u30a4\u30ba\u306f\u5b9f\u904b\u7528\u6642\u306b\u306f\u8981\u8abf\u6574)<\/p>\n<p>spark-<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/sql\">sql<\/a>\u3067\u5b9f\u884c\u3059\u308b\u3002(\u5b9f\u969b\u306e\u30c6\u30fc\u30d6\u30eb\u5b9a\u7fa9\u304b\u3089\u7c21\u6613\u5316\u3057\u3001\u307e\u305f\u7c21\u5358\u306e\u305f\u3081\u306b\u4e8b\u524d\u306e\u5909\u63db\u30aa\u30da\u30ec\u30fc\u30b7\u30e7\u30f3\u306a\u3069\u3082\u7701\u7565\u3057\u3066\u3042\u308a\u307e\u3059)<\/p>\n<pre class=\"code lang-sql\" data-lang=\"sql\" data-unlink=\"\"><span class=\"synStatement\">CREATE<\/span> <span class=\"synSpecial\">TABLE<\/span> <span class=\"synSpecial\">IF<\/span> <span class=\"synStatement\">NOT<\/span> <span class=\"synStatement\">EXISTS<\/span> glue.testdb_production.test_tables (\n  app_id bigint <span class=\"synStatement\">NOT<\/span> <span class=\"synSpecial\">NULL<\/span>,\n  user_id bigint <span class=\"synStatement\">NOT<\/span> <span class=\"synSpecial\">NULL<\/span>,\n  key string <span class=\"synStatement\">NOT<\/span> <span class=\"synSpecial\">NULL<\/span>,\n  value string\n)\n<span class=\"synSpecial\">USING<\/span> iceberg\nPARTITIONED <span class=\"synSpecial\">BY<\/span> (bucket(<span class=\"synConstant\">32<\/span>, app_id), bucket(<span class=\"synConstant\">8<\/span>, key))\nTBLPROPERTIES (\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">write.object-storage.enabled<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">true<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">write.delete.mode<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">merge-on-read<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">write.update.mode<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">merge-on-read<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">write.merge.mode<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">merge-on-read<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">history.expire.max-snapshot-age-ms<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">86400000<\/span><span class=\"synSpecial\">'<\/span>\n)\nLOCATION <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">s3:\/\/repro-experimental-store\/production\/testdb_production\/test_tables<\/span><span class=\"synSpecial\">'<\/span>;\n\n<span class=\"synStatement\">ALTER<\/span> <span class=\"synSpecial\">TABLE<\/span> glue.testdb_production.test_tables WRITE ORDERED <span class=\"synSpecial\">BY<\/span> insight_id, key;\n\n<span class=\"synStatement\">ALTER<\/span> <span class=\"synSpecial\">TABLE<\/span> glue.testdb_production.test_tables <span class=\"synStatement\">SET<\/span> IDENTIFIER FIELDS insight_id, user_id, key;\n<\/pre>\n<p>Flink\u3067\u9ad8\u983b\u5ea6\u306e\u66f8\u304d\u8fbc\u3080\u3053\u3068\u3067snapshot\u304c\u5927\u91cf\u306b\u5897\u3048\u308b\u305f\u3081max-snapshot-age-ms\u3092\u77ed\u304f\u8a2d\u5b9a\u3057\u3001merge-on-read\u3068object storage\u5411\u3051\u306e\u6700\u9069\u5316(prefix\u3092\u3070\u3089\u3051\u3055\u305b\u308b\u3053\u3068\u3067\u66f8\u304d\u8fbc\u307f<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%B9%A5%EB%A1%BC%A5%D7%A5%C3%A5%C8\">\u30b9\u30eb\u30fc\u30d7\u30c3\u30c8<\/a>\u306e\u30ad\u30e3\u30c3\u30d7\u3092\u56de\u907f\u3059\u308b)\u3092\u6709\u52b9\u5316\u3002<br \/>\n\u305d\u306e\u4ed6\u306e\u30d5\u30a1\u30a4\u30eb\u30b5\u30a4\u30ba\u306e\u57fa\u6e96\u5024\u306a\u3069\u306f\u30c7\u30d5\u30a9\u30eb\u30c8\u3092\u5229\u7528\u3002<\/p>\n<h3 id=\"Iceberg\u306e\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30cb\u30f3\u30b0\u306b\u3064\u3044\u3066\u88dc\u8db3\">Iceberg\u306e\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30cb\u30f3\u30b0\u306b\u3064\u3044\u3066\u88dc\u8db3<\/h3>\n<p>Iceberg\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u60c5\u5831\u306f\u65e7\u6765\u306eHive\u5f62\u5f0f\u3068\u306f\u9055\u3063\u3066<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%E1%A5%BF%A5%C7%A1%BC%A5%BF\">\u30e1\u30bf\u30c7\u30fc\u30bf<\/a>\u30d5\u30a1\u30a4\u30eb\u306e\u4e2d\u306b\u8a18\u9332\u3055\u308c\u3066\u3044\u307e\u3059\u3002<br \/>\n\u305d\u3057\u3066\u3001<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%E1%A5%BF%A5%C7%A1%BC%A5%BF\">\u30e1\u30bf\u30c7\u30fc\u30bf<\/a>\u306f\u5c65\u6b74\u7ba1\u7406\u304c\u53ef\u80fd\u306a\u306e\u3067Iceberg\u306f\u30c7\u30fc\u30bf\u30d5\u30a1\u30a4\u30eb\u306e\u914d\u7f6e\u3092\u5909\u66f4\u3057\u306a\u304f\u3066\u3082\u9014\u4e2d\u304b\u3089\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30cb\u30f3\u30b0\u306e\u5f62\u5f0f\u3092\u5909\u66f4\u3059\u308b\u3053\u3068\u304c\u3067\u304d\u307e\u3059\u3002<br \/>\n\u307e\u305f\u3001Bucketing\u306e\u305f\u3081\u306e\u95a2\u6570\u3084\u30bf\u30a4\u30e0\u30b9\u30bf\u30f3\u30d7\u304b\u3089\u5e74\u30fb\u6708\u30fb\u65e5\u306b\u5206\u89e3\u3059\u308b\u95a2\u6570\u3092\u565b\u307e\u305b\u3066\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30cb\u30f3\u30b0\u3059\u308b\u3053\u3068\u3082\u53ef\u80fd\u3067\u3001\u30af\u30a8\u30ea\u6642\u306b\u306f\u305d\u308c\u3092\u5143\u306b\u306a\u3063\u305f\u30ab\u30e9\u30e0\u306e\u62bd\u51fa\u6761\u4ef6\u304b\u3089\u900f\u904e\u7684\u306b\u51e6\u7406\u3057\u3066\u304f\u308c\u307e\u3059\u3002<br \/>\n\u3053\u306e\u4ed5\u7d44\u307f\u3092hidden partition\u3068\u547c\u3073\u307e\u3059\u3002<\/p>\n<p>\u4e0a\u8a18\u306e\u30c6\u30fc\u30d6\u30eb\u5b9a\u7fa9\u3067\u306f\u60f3\u5b9a\u3055\u308c\u308b\u30af\u30a8\u30ea\u306e\u30d1\u30bf\u30fc\u30f3\u306b\u5408\u308f\u305b\u3066\u4e8c\u3064\u306e\u30ad\u30fc\u3067bucketing\u3059\u308b\u5f62\u306b\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<h2 id=\"EMR\u8a2d\u5b9a\">EMR\u8a2d\u5b9a<\/h2>\n<p>\u4ee5\u4e0b\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u30b5\u30a4\u30ba\u3001\u8a2d\u5b9a\u5185\u5bb9\u306f\u8a66\u884c\u932f\u8aa4\u306e\u7d50\u679c\u843d\u3061\u7740\u3044\u305f\u5024\u3067\u3001\u5b9f\u969b\u306e\u4f5c\u696d\u4e2d\u306f\u52d5\u4f5c\u3092\u898b\u3064\u3064\u4f55\u5ea6\u3082\u30c6\u30fc\u30d6\u30eb\u3092\u4f5c\u308a\u76f4\u3057\u3066\u53f0\u6570\u3084\u8a2d\u5b9a\u5024\u3092\u8abf\u6574\u3057\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u66f8\u304d\u8fbc\u307f<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u30b5\u30a4\u30ba: r8g.2xlarge * 3<br \/>\n\u30c6\u30fc\u30d6\u30eb\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u7528<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u30b5\u30a4\u30ba: r8g.2xlarge * 20 (\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u30bf\u30b9\u30af\u5b9f\u884c\u6642\u306e\u307f\u5fc5\u8981)<\/p>\n<p>Config:<\/p>\n<pre class=\"code lang-json\" data-lang=\"json\" data-unlink=\"\"><span class=\"synSpecial\">[<\/span>\n  <span class=\"synSpecial\">{<\/span>\n    \"<span class=\"synStatement\">Classification<\/span>\": \"<span class=\"synConstant\">iceberg-defaults<\/span>\",\n    \"<span class=\"synStatement\">Properties<\/span>\": <span class=\"synSpecial\">{<\/span>\n      \"<span class=\"synStatement\">iceberg.enabled<\/span>\": \"<span class=\"synConstant\">true<\/span>\"\n    <span class=\"synSpecial\">}<\/span>\n  <span class=\"synSpecial\">}<\/span>,\n  <span class=\"synSpecial\">{<\/span>\n    \"<span class=\"synStatement\">Classification<\/span>\": \"<span class=\"synConstant\">spark<\/span>\",\n    \"<span class=\"synStatement\">Properties<\/span>\": <span class=\"synSpecial\">{<\/span>\n      \"<span class=\"synStatement\">maximizeResourceAllocation<\/span>\": \"<span class=\"synConstant\">true<\/span>\"\n    <span class=\"synSpecial\">}<\/span>\n  <span class=\"synSpecial\">}<\/span>,\n  <span class=\"synSpecial\">{<\/span>\n    \"<span class=\"synStatement\">Classification<\/span>\": \"<span class=\"synConstant\">spark-defaults<\/span>\",\n    \"<span class=\"synStatement\">Properties<\/span>\": <span class=\"synSpecial\">{<\/span>\n      \"<span class=\"synStatement\">spark.driver.memory<\/span>\": \"<span class=\"synConstant\">8g<\/span>\",\n      \"<span class=\"synStatement\">spark.serializer<\/span>\": \"<span class=\"synConstant\">org.apache.spark.serializer.KryoSerializer<\/span>\",\n      \"<span class=\"synStatement\">spark.executor.memory<\/span>\": \"<span class=\"synConstant\">44g<\/span>\",\n      \"<span class=\"synStatement\">spark.memory.fraction<\/span>\": \"<span class=\"synConstant\">0.2<\/span>\",\n      \"<span class=\"synStatement\">spark.memory.storageFraction<\/span>\": \"<span class=\"synConstant\">0.3<\/span>\",\n      \"<span class=\"synStatement\">spark.executor.extraJavaOptions<\/span>\": \"<span class=\"synConstant\">-XX:+UseG1GC -XX:MaxGCPauseMillis=1500 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1ReservePercent=15 -XX:ConcGCThreads=6<\/span>\",\n      \"<span class=\"synStatement\">spark.task.cpus<\/span>\": \"<span class=\"synConstant\">8<\/span>\",\n      \"<span class=\"synStatement\">spark.sql.catalog.glue<\/span>\": \"<span class=\"synConstant\">org.apache.iceberg.spark.SparkCatalog<\/span>\",\n      \"<span class=\"synStatement\">spark.sql.catalog.glue.warehouse<\/span>\": \"<span class=\"synConstant\">s3:\/\/aws-glue-repro-iceberg-test2\/iceberg_test<\/span>\",\n      \"<span class=\"synStatement\">spark.sql.catalog.glue.type<\/span>\": \"<span class=\"synConstant\">glue<\/span>\",\n      \"<span class=\"synStatement\">spark.sql.defaultCatalog<\/span>\": \"<span class=\"synConstant\">glue<\/span>\",\n      \"<span class=\"synStatement\">spark.sql.extensions<\/span>\": \"<span class=\"synConstant\">org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions<\/span>\",\n      \"<span class=\"synStatement\">spark.sql.catalog.glue.io-impl<\/span>\": \"<span class=\"synConstant\">org.apache.iceberg.aws.s3.S3FileIO<\/span>\"\n    <span class=\"synSpecial\">}<\/span>\n  <span class=\"synSpecial\">}<\/span>,\n  <span class=\"synSpecial\">{<\/span>\n    \"<span class=\"synStatement\">Classification<\/span>\": \"<span class=\"synConstant\">flink-conf<\/span>\",\n    \"<span class=\"synStatement\">Properties<\/span>\": <span class=\"synSpecial\">{<\/span>\n      \"<span class=\"synStatement\">glue.enabled<\/span>\": \"<span class=\"synConstant\">true<\/span>\",\n      \"<span class=\"synStatement\">jobmanager.memory.process.size<\/span>\": \"<span class=\"synConstant\">2gb<\/span>\",\n      \"<span class=\"synStatement\">kafka.enabled<\/span>\": \"<span class=\"synConstant\">true<\/span>\",\n      \"<span class=\"synStatement\">taskmanager.memory.managed.fraction<\/span>\": \"<span class=\"synConstant\">0.3<\/span>\",\n      \"<span class=\"synStatement\">taskmanager.memory.process.size<\/span>\": \"<span class=\"synConstant\">40gb<\/span>\",\n      \"<span class=\"synStatement\">env.java.opts.taskmanager<\/span>\": \"-<span class=\"synError\">Xlog<\/span>:<span class=\"synError\">gc<\/span> -<span class=\"synError\">XX<\/span>:+<span class=\"synError\">UseG1GC<\/span> -<span class=\"synError\">XX<\/span>:<span class=\"synError\">MaxGCPauseMillis<\/span>=1000 -<span class=\"synError\">XX<\/span>:<span class=\"synError\">InitiatingHeapOccupancyPercent<\/span>=25 -<span class=\"synError\">XX<\/span>:<span class=\"synError\">G1NewSizePercent<\/span>=15\"\n      \"<span class=\"synStatement\">job.autoscaler.enabled<\/span>\": \"<span class=\"synConstant\">true<\/span>\",\n      \"<span class=\"synStatement\">jobmanager.scheduler<\/span>\": \"<span class=\"synConstant\">adaptive<\/span>\",\n      \"<span class=\"synStatement\">job.autoscaler.stabilization.interval<\/span>\": \"<span class=\"synConstant\">300s<\/span>\",\n      \"<span class=\"synStatement\">job.autoscaler.metrics.window<\/span>\": \"<span class=\"synConstant\">600s<\/span>\",\n      \"<span class=\"synStatement\">job.autoscaler.decision.interval<\/span>\": \"<span class=\"synConstant\">60s<\/span>\"\n    <span class=\"synSpecial\">}<\/span>\n  <span class=\"synSpecial\">}<\/span>,\n  <span class=\"synSpecial\">{<\/span>\n    \"<span class=\"synStatement\">Classification<\/span>\": \"<span class=\"synConstant\">trino-connector-cassandra<\/span>\",\n    \"<span class=\"synStatement\">Properties<\/span>\": <span class=\"synSpecial\">{<\/span>\n      \"<span class=\"synStatement\">cassandra.client.read-timeout<\/span>\": \"<span class=\"synConstant\">5m<\/span>\",\n      \"<span class=\"synStatement\">cassandra.consistency-level<\/span>\": \"<span class=\"synConstant\">LOCAL_QUORUM<\/span>\",\n      \"<span class=\"synStatement\">cassandra.contact-points<\/span>\": \"<span class=\"synConstant\"><cassandra server=\"\" ips=\"\"\/><\/span>\",\n      \"<span class=\"synStatement\">cassandra.fetch-size<\/span>\": \"<span class=\"synConstant\">1000<\/span>\",\n      \"<span class=\"synStatement\">cassandra.load-policy.dc-aware.local-dc<\/span>\": \"<span class=\"synConstant\">ap-northeast_3az<\/span>\",\n      \"<span class=\"synStatement\">cassandra.load-policy.use-dc-aware<\/span>\": \"<span class=\"synConstant\">true<\/span>\",\n      \"<span class=\"synStatement\">cassandra.partition-size-for-batch-select<\/span>\": \"<span class=\"synConstant\">256<\/span>\",\n      \"<span class=\"synStatement\">cassandra.password<\/span>\": \"<span class=\"synConstant\">******<\/span>\",\n      \"<span class=\"synStatement\">cassandra.security<\/span>\": \"<span class=\"synConstant\">PASSWORD<\/span>\",\n      \"<span class=\"synStatement\">cassandra.username<\/span>\": \"<span class=\"synConstant\">cassandra<\/span>\",\n      \"<span class=\"synStatement\">connector.name<\/span>\": \"<span class=\"synConstant\">cassandra<\/span>\"\n    <span class=\"synSpecial\">}<\/span>\n  <span class=\"synSpecial\">}<\/span>,\n  <span class=\"synSpecial\">{<\/span>\n    \"<span class=\"synStatement\">Classification<\/span>\": \"<span class=\"synConstant\">trino-connector-kafka<\/span>\",\n    \"<span class=\"synStatement\">Properties<\/span>\": <span class=\"synSpecial\">{<\/span>\n      \"<span class=\"synStatement\">connector.name<\/span>\": \"<span class=\"synConstant\">kafka<\/span>\",\n      \"<span class=\"synStatement\">kafka.confluent-schema-registry-url<\/span>\": \"<span class=\"synConstant\"><schema registry=\"\" urls=\"\"\/><\/span>\",\n      \"<span class=\"synStatement\">kafka.nodes<\/span>\": \"<span class=\"synConstant\"><kafka broker=\"\" ips=\"\"\/><\/span>\",\n      \"<span class=\"synStatement\">kafka.table-description-supplier<\/span>\": \"<span class=\"synConstant\">CONFLUENT<\/span>\"\n    <span class=\"synSpecial\">}<\/span>\n  <span class=\"synSpecial\">}<\/span>,\n  <span class=\"synSpecial\">{<\/span>\n    \"<span class=\"synStatement\">Classification<\/span>\": \"<span class=\"synConstant\">trino-connector-iceberg<\/span>\",\n    \"<span class=\"synStatement\">Properties<\/span>\": <span class=\"synSpecial\">{<\/span>\n      \"<span class=\"synStatement\">connector.name<\/span>\": \"<span class=\"synConstant\">iceberg<\/span>\",\n      \"<span class=\"synStatement\">fs.native-s3.enabled<\/span>\": \"<span class=\"synConstant\">true<\/span>\",\n      \"<span class=\"synStatement\">iceberg.catalog.type<\/span>\": \"<span class=\"synConstant\">glue<\/span>\",\n      \"<span class=\"synStatement\">iceberg.max-partitions-per-writer<\/span>\": \"<span class=\"synConstant\">1024<\/span>\",\n      \"<span class=\"synStatement\">iceberg.object-store-layout.enabled<\/span>\": \"<span class=\"synConstant\">true<\/span>\"\n    <span class=\"synSpecial\">}<\/span>\n  <span class=\"synSpecial\">}<\/span>\n<span class=\"synSpecial\">]<\/span>\n<\/pre>\n<h2 id=\"\u521d\u671f\u69cb\u7bc9\">\u521d\u671f\u69cb\u7bc9<\/h2>\n<p>Flink\u3067\u66f8\u304d\u8fbc\u307f\u3092\u884c\u3046\u524d\u306b\u3001\u65e2\u5b58\u306e\u30c7\u30fc\u30bf\u3092\u5143\u306b\u521d\u671f\u30c7\u30fc\u30bf\u3092\u6295\u5165\u3057\u672c\u756a\u3068\u540c\u7b49\u306e\u30b5\u30a4\u30ba\u306e\u30c6\u30fc\u30d6\u30eb\u3092\u69cb\u7bc9\u3059\u308b\u3002<\/p>\n<p>\u65e2\u5b58\u306e\u30c7\u30fc\u30bf\u306fcassandra\u306b\u84c4\u7a4d\u3055\u308c\u3066\u3044\u308b\u305f\u3081\u3001\u3053\u308c\u3092\u5909\u63db\u3057\u3066Trino\u7d4c\u7531\u3067iceberg\u30c6\u30fc\u30d6\u30eb\u306b\u6295\u5165\u3059\u308b\u3002<\/p>\n<pre class=\"code lang-sql\" data-lang=\"sql\" data-unlink=\"\"><span class=\"synStatement\">INSERT<\/span> <span class=\"synSpecial\">INTO<\/span> iceberg.testdb_production.test_tables (app_id, user_id, key, value)\n<span class=\"synStatement\">SELECT<\/span> app_id, user_id, key, value <span class=\"synSpecial\">FROM<\/span> cassandra.repro.test_tables;\n<\/pre>\n<p>\u4e0a\u8a18\u306e\u30c7\u30fc\u30bf\u898f\u6a21\u3092\u6295\u5165\u3059\u308b\u306e\u306br8g.2xlarge 8\u53f0\u306etrino<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u30671\u6642\u9593\u534a\u3050\u3089\u3044\u306e\u6240\u8981\u6642\u9593\u3067\u5b8c\u4e86\u3059\u308b\u3002<\/p>\n<h2 id=\"Flink\u306b\u3088\u308b\u66f8\u304d\u8fbc\u307f\u8a2d\u5b9a\">Flink\u306b\u3088\u308b\u66f8\u304d\u8fbc\u307f\u8a2d\u5b9a<\/h2>\n<p>confluent\u306eschema registry\u3092\u5229\u7528\u3057\u305fKafka\u30c8\u30d4\u30c3\u30af\u304b\u3089\u306e\u30b9\u30c8\u30ea\u30fc\u30e0\u66f8\u304d\u8fbc\u307f\u3092\u884c\u3046\u305f\u3081\u3001\u4ee5\u4e0b\u306e\u6e96\u5099\u3092\u3059\u308b\u3002<\/p>\n<pre class=\"code lang-sh\" data-lang=\"sh\" data-unlink=\"\">\nwget https:\/\/repo.maven.apache.org\/maven2\/org\/apache\/flink\/flink-sql-avro-confluent-registry\/<span class=\"synConstant\">1<\/span>.<span class=\"synConstant\">20<\/span>.<span class=\"synConstant\">0<\/span>\/flink-sql-avro-confluent-registry-1.<span class=\"synConstant\">20<\/span>.<span class=\"synConstant\">0<\/span>.jar\n<\/pre>\n<pre class=\"code lang-sh\" data-lang=\"sh\" data-unlink=\"\">\nflink-yarn-session <span class=\"synSpecial\">-Dparallelism<\/span>.<span class=\"synIdentifier\">default<\/span>=<span class=\"synConstant\">2<\/span> <span class=\"synSpecial\">-d<\/span>\n<\/pre>\n<pre class=\"code lang-sh\" data-lang=\"sh\" data-unlink=\"\">\nflink-sql-client <span class=\"synSpecial\">-j<\/span> flink-sql-avro-confluent-registry-1.<span class=\"synConstant\">20<\/span>.<span class=\"synConstant\">0<\/span>.jar <span class=\"synSpecial\">-f<\/span> insert.sql\n<\/pre>\n<pre class=\"code lang-sql\" data-lang=\"sql\" data-unlink=\"\"><span class=\"synStatement\">SET<\/span> state.backend.<span class=\"synSpecial\">type<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">rocksdb<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.storage = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">filesystem<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.dir = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">s3:\/\/repro-experimental-store\/production\/flink-checkpoints<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.<span class=\"synStatement\">savepoint<\/span>-dir = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">s3:\/\/repro-experimental-store\/production\/flink-checkpoints<\/span><span class=\"synSpecial\">'<\/span>;\n\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.num-retained = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">1<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.interval = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">15min<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.timeout = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">10min<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.<span class=\"synIdentifier\">min<\/span>-pause = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">1min<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.externalized-checkpoint-retention = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">RETAIN_ON_CANCELLATION<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> execution.checkpointing.<span class=\"synIdentifier\">min<\/span>-pause = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">3min<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> state.backend.incremental = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">true<\/span><span class=\"synSpecial\">'<\/span>;\n\n<span class=\"synStatement\">SET<\/span> <span class=\"synSpecial\">table<\/span>.exec.mini-batch.enabled = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">true<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> <span class=\"synSpecial\">table<\/span>.exec.mini-batch.allow-latency = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">10s<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> <span class=\"synSpecial\">table<\/span>.exec.mini-batch.<span class=\"synSpecial\">size<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">10000<\/span><span class=\"synSpecial\">'<\/span>;\n<span class=\"synStatement\">SET<\/span> <span class=\"synSpecial\">table<\/span>.exec.deduplicate.mini-batch.compact-changes-enabled = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">true<\/span><span class=\"synSpecial\">'<\/span>;\n\n<span class=\"synStatement\">CREATE<\/span> CATALOG glue_catalog <span class=\"synSpecial\">WITH<\/span> (\n   <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">type<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">iceberg<\/span><span class=\"synSpecial\">'<\/span>,\n   <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">warehouse<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">s3:\/\/aws-glue-repro-iceberg-test\/iceberg_test<\/span><span class=\"synSpecial\">'<\/span>,\n   <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">catalog-impl<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">org.apache.iceberg.aws.glue.GlueCatalog<\/span><span class=\"synSpecial\">'<\/span>,\n   <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">io-impl<\/span><span class=\"synSpecial\">'<\/span>=<span class=\"synSpecial\">'<\/span><span class=\"synConstant\">org.apache.iceberg.aws.s3.S3FileIO<\/span><span class=\"synSpecial\">'<\/span>\n );\n\n<span class=\"synStatement\">CREATE<\/span> <span class=\"synSpecial\">TABLE<\/span> <span class=\"synSpecial\">IF<\/span> <span class=\"synStatement\">NOT<\/span> <span class=\"synStatement\">EXISTS<\/span> test_tables_kafka (\n  \n  kafka_key STRING <span class=\"synStatement\">NOT<\/span> <span class=\"synSpecial\">NULL<\/span>,\n\n  insight_id <span class=\"synType\">INT<\/span> <span class=\"synStatement\">NOT<\/span> <span class=\"synSpecial\">NULL<\/span>,\n  user_id BIGINT <span class=\"synStatement\">NOT<\/span> <span class=\"synSpecial\">NULL<\/span>,\n  key STRING <span class=\"synStatement\">NOT<\/span> <span class=\"synSpecial\">NULL<\/span>,\n  ts <span class=\"synType\">TIMESTAMP<\/span>(<span class=\"synConstant\">3<\/span>) METADATA <span class=\"synSpecial\">FROM<\/span> <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">timestamp<\/span><span class=\"synSpecial\">'<\/span>,\n  WATERMARK <span class=\"synSpecial\">FOR<\/span> ts <span class=\"synSpecial\">AS<\/span> ts - INTERVAL <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">5<\/span><span class=\"synSpecial\">'<\/span> SECOND\n) <span class=\"synSpecial\">WITH<\/span> (\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">connector<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">kafka<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">topic<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">TestTopic<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">properties.bootstrap.servers<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\"><kafka broker=\"\" ips=\"\"\/><\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">properties.group.id<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">flink-iceberg-experiment<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">scan.startup.mode<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">group-offsets<\/span><span class=\"synSpecial\">'<\/span>,\n  \n  \n\n  \n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">key.format<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">raw<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">key.fields<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">kafka_key<\/span><span class=\"synSpecial\">'<\/span>,\n\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">value.format<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">avro-confluent<\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">value.avro-confluent.url<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\"><schema registry=\"\" url=\"\"\/><\/span><span class=\"synSpecial\">'<\/span>,\n  <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">value.fields-include<\/span><span class=\"synSpecial\">'<\/span> = <span class=\"synSpecial\">'<\/span><span class=\"synConstant\">EXCEPT_KEY<\/span><span class=\"synSpecial\">'<\/span>\n);\n\n\n<span class=\"synStatement\">INSERT<\/span> <span class=\"synSpecial\">INTO<\/span> glue_catalog.testdb_production.test_tables  (app_id, user_id, key, value)\n<span class=\"synStatement\">SELECT<\/span> insight_id, user_id, key, value <span class=\"synSpecial\">FROM<\/span> default_catalog.default_database.test_tables_kafka;\n<\/pre>\n<p>\u7279\u306b\u91cd\u8981\u306a\u8a2d\u5b9a\u306f\u4ee5\u4e0b\u3002<\/p>\n<ul>\n<li><code>execution.checkpointing.interval<\/code>: checkpoint\u306e\u5b9f\u884c\u9593\u9694\u304ciceberg\u306ecommit\u9593\u9694\u306b\u306a\u308b\u3002\u4eca\u56de\u306f\u8abf\u6574\u306e\u7d50\u679c15\u5206\u3068\u3057\u305f\u3002<\/li>\n<li>INSERT\u6587\u306e<code>OPTIONS('upsert-enabled'='true')<\/code>: iceberg\u306e\u66f8\u304d\u8fbc\u307f\u306b\u969b\u3057\u3066upsert\u30e2\u30fc\u30c9\u3067\u66f8\u304d\u8fbc\u307f\u3092\u884c\u3046\u8a2d\u5b9a\u3002flink <a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/SQL\">SQL<\/a>\u306e\u30d2\u30f3\u30c8\u53e5\u3068\u3057\u3066\u8a2d\u5b9a\u3057\u3066\u3044\u308b\u3002<\/li>\n<li><code>scan.startup.mode<\/code>: kafka\u30c8\u30d4\u30c3\u30af\u306e\u3069\u3053\u304b\u3089\u53d6\u5f97\u3092\u958b\u59cb\u3059\u308b\u306e\u304b\u3092\u8a2d\u5b9a\u3059\u308b\u3002\u521d\u671f\u69cb\u7bc9\u3092trino\u3067\u5b9f\u65bd\u3057\u305f\u306e\u3067\u3001\u521d\u8d77\u52d5\u6642\u306f\u305d\u308c\u306b\u5408\u308f\u305b\u3066\u5c11\u3057\u524d\u306etimestamp\u304b\u3089\u5b9f\u884c\u3059\u308b\u3002\u4e00\u5ea6group.id\u306eoffset\u304ckafka broker\u306b\u8a18\u9332\u3055\u308c\u305f\u3089\u79fb\u884c\u306f<code>group-offsets<\/code>\u3092\u5229\u7528\u3057\u3066\u305d\u3053\u304b\u3089\u51e6\u7406\u3092\u518d\u958b\u3067\u304d\u308b\u3002<\/li>\n<\/ul>\n<p>\u307e\u305f\u3001EMR\u5074\u306e\u8a2d\u5b9a\u3067flink\u306eadaptive\u30b9\u30b1\u30b8\u30e5\u30fc\u30e9\u30fc\u3092\u6709\u52b9\u306b\u3057\u3066\u3044\u3066\u3001\u8ca0\u8377\u306b\u5408\u308f\u305b\u3066\u81ea\u52d5\u3067\u30bf\u30b9\u30af\u6570\u304c\u5897\u6e1b\u3059\u308b\u3002<br \/>\n\u73fe\u72b6\u306e\u79d2\u95935000\u4ef6\u7a0b\u5ea6\u306e\u66f4\u65b0\u8981\u6c42\u3067\u3042\u308c\u30701\u53f0\u3067\u4f59\u88d5\u3067\u51e6\u7406\u3067\u304d\u308b\u3053\u3068\u304c\u5206\u304b\u3063\u305f\u3002<\/p>\n<p>1\u30bf\u30b9\u30af\u306715\u5206\u306b1\u56de\u30b3\u30df\u30c3\u30c8\u306a\u306e\u3067\u300115\u5206\u306b\u51e1\u305d1<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u306b1\u30d5\u30a1\u30a4\u30eb\u3065\u3064parquet\u30d5\u30a1\u30a4\u30eb\u304c\u5897\u3048\u3066\u3044\u304f\u3002<br \/>\n24\u6642\u9593\u3067 4(1h\u30674\u56de) * 24h * 2(data file &amp; delete file) = 96\u30d5\u30a1\u30a4\u30eb\u7a0b\u5897\u3048\u308b\u3053\u3068\u306b\u306a\u308b\u3002<br \/>\n\u4e26\u5217\u6570\u304c\u3082\u3063\u3068\u5fc5\u8981\u3067\u3042\u308c\u3070\u3001\u305d\u306e\u5206\u4f5c\u6210\u3055\u308c\u308b\u30d5\u30a1\u30a4\u30eb\u6570\u3082\u5897\u3048\u308b\u3002<\/p>\n<p>\u3053\u308c\u3092\u5b9a\u671f\u7684\u306acompaction\u3067\u89e3\u6d88\u3067\u304d\u308b\u304b\u3069\u3046\u304b\u3092\u691c\u8a3c\u3057\u305f\u3002<\/p>\n<h2 id=\"\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u30bf\u30b9\u30af\u306e\u5b9f\u884c\">\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u30bf\u30b9\u30af\u306e\u5b9f\u884c<\/h2>\n<p>\u691c\u8a3c\u958b\u59cb\u5f53\u521d\u306fGlue Data Catalog\u306eTable Optimization\u306b\u4efb\u305b\u308b\u3064\u3082\u308a\u3060\u3063\u305f\u304c\u3001compaction\u306e\u6319\u52d5\u304c\u3069\u3046\u3082\u671f\u5f85\u901a\u308a\u3067\u306f\u306a\u304b\u3063\u305f\u306e\u3067\u3001expire_snapshot\u3068delete_orpharn_files\u3060\u3051Glue\u306e\u6a5f\u80fd\u306b\u4efb\u305b\u3066\u3001compaction\u306fEMR\u3092\u5b9a\u671f\u7684\u306b\u5b9f\u884c\u3059\u308b\u5f62\u306b\u3057\u305f\u3002<\/p>\n<p>Glue Data Catalog\u306eTable Optimization\u306f\u5b9f\u884c\u81ea\u4f53\u306f\u3061\u3083\u3093\u3068\u52d5\u304f\u306e\u3060\u304c\u3001\u4e0a\u8a18\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/DDL\">DDL<\/a>\u3067\u4f5c\u6210\u3057\u305f<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u306e\u5168\u3066\u306b\u5bfe\u3057\u3066compaction\u304c\u884c\u308f\u308c\u3066\u3044\u306a\u3044\u69d8\u306a\u6319\u52d5\u3092\u89b3\u6e2c\u3057\u305f\u3002\u3053\u308c\u3067\u306f\u975e\u5e38\u306b\u56f0\u308b\u306e\u3067\u4e00\u65e6\u5229\u7528\u3092\u8ae6\u3081\u3066\u73fe\u5728<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/AWS\">AWS<\/a>\u306e\u30b5\u30dd\u30fc\u30c8\u306b\u554f\u3044\u5408\u308f\u305b\u4e2d\u3002<\/p>\n<p>expire_snapshot\u306f12\u6642\u9593\u306b1\u56de\u5b9f\u884c\u3001delete_orphan_files\u306f24\u6642\u9593\u306b1\u56de\u5b9f\u884c\u305748\u6642\u9593\u3088\u308a\u524d\u306e\u30d5\u30a1\u30a4\u30eb\u3092\u524a\u9664\u3059\u308b\u7528\u306b\u8a2d\u5b9a\u3057\u305f\u3002<\/p>\n<p>compaction\u306fEMR\u306ecommand-runner\u30b9\u30c6\u30c3\u30d7\u3092\u5229\u7528\u3057\u306624\u6642\u9593\u306b1\u56de\u4ee5\u4e0b\u306eshell\u30b3\u30de\u30f3\u30c9\u3092\u5b9f\u884c\u3059\u308b\u3002<\/p>\n<pre class=\"code lang-sh\" data-lang=\"sh\" data-unlink=\"\">spark-sql <span class=\"synSpecial\">-e<\/span> <span class=\"synStatement\">\"<\/span><span class=\"synConstant\">CALL glue.system.rewrite_data_files(table =&gt; 'testdb_production.test_tables', strategy =&gt; 'sort', options =&gt; map('min-input-files', '50', 'partial-progress.enabled', 'true', 'partial-progress.max-commits', '10', 'max-concurrent-file-group-rewrites', '3', 'remove-dangling-deletes', 'true', 'rewrite-job-order', 'files-desc'));<\/span><span class=\"synStatement\">\"<\/span>\n<\/pre>\n<h3 id=\"compaction\u30aa\u30d7\u30b7\u30e7\u30f3\u89e3\u8aac\">compaction\u30aa\u30d7\u30b7\u30e7\u30f3\u89e3\u8aac<\/h3>\n<ul>\n<li><code>min-input-files<\/code>:  \u3042\u308b<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u306e\u30d5\u30a1\u30a4\u30eb\u6570\u304c\u3053\u308c\u3092\u8d8a\u3048\u3066\u305f\u3089\u5bfe\u8c61\u3068\u3059\u308b\u3002<\/li>\n<li><code>partial-progress.enabled<\/code>: \u9032\u6357\u306e\u9014\u4e2d\u3067\u306e\u30b3\u30df\u30c3\u30c8\u3092\u6709\u52b9\u306b\u3059\u308b\u3002\u5168\u3066\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u306e\u51e6\u7406\u304c\u7d42\u4e86\u3059\u308b\u524d\u306b\u51e6\u7406\u72b6\u6cc1\u3092\u53cd\u6620\u3067\u304d\u308b\u3002\u3069\u308c\u3050\u3089\u3044\u306e\u983b\u5ea6\u3067\u30b3\u30df\u30c3\u30c8\u3059\u308b\u304b\u306f<code>partial-progress.max-commits<\/code>\u3068\u30bf\u30b9\u30af\u7dcf\u6570\u306b\u3088\u3063\u3066\u6c7a\u307e\u308b\u3002<\/li>\n<li><code>max-concurrent-file-group-rewrites<\/code>: \u540c\u6642\u306b\u51e6\u7406\u3059\u308b\u30d5\u30a1\u30a4\u30eb\u30b0\u30eb\u30fc\u30d7(<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>)\u30bf\u30b9\u30af\u306e\u6570\u3002<\/li>\n<li><code>remove-dangling-deletes<\/code>: compaction\u306e\u7d50\u679c\u5fc5\u8981\u306a\u304f\u306a\u3063\u305fdelete\u30d5\u30a1\u30a4\u30eb\u3092\u524a\u9664\u3059\u308b\u3002<\/li>\n<li><code>rewrite-job-order<\/code>: <a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u3054\u3068\u306e\u51e6\u7406\u9806\u756a\u3092\u6c7a\u3081\u308b\u3002<code>files-desc<\/code>\u306f\u30d5\u30a1\u30a4\u30eb\u7dcf\u6570\u306e\u591a\u3044\u9806\u306b\u51e6\u7406\u3002<\/li>\n<\/ul>\n<p>24\u6642\u9593\u30671<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u3054\u3068\u306b\u6700\u4f4e\u3067\u308296\u30d5\u30a1\u30a4\u30eb + deletes\u30d5\u30a1\u30a4\u30eb\u5206\u306e\u5c0f\u3055\u3044\u30d5\u30a1\u30a4\u30eb\u304c\u751f\u6210\u3055\u308c\u308b\u306e\u3067\u3001\u305d\u308c\u3092compaction\u306b\u3088\u308a\u5927\u304d\u306aparquet\u30d5\u30a1\u30a4\u30eb\u306b\u7d50\u5408\u3059\u308b\u3002<br \/>\n\u3053\u306e\u30d5\u30a1\u30a4\u30eb\u5897\u52a0\u30da\u30fc\u30b9\u3067\u3042\u308c\u3070\u3001\u5927\u4f53r8g.2xlarge * 20\u53f0\u30671\u6642\u9593\u301c2\u6642\u9593\u3050\u3089\u3044\u306e\u51e6\u7406\u6642\u9593\u3067\u30bf\u30b9\u30af\u304c\u5b8c\u4e86\u3059\u308b\u3002<\/p>\n<p>24\u6642\u9593\u306b1\u5ea6\u30012\u6642\u9593\u306e\u6240\u8981\u6642\u9593\u306e\u5b9f\u884c\u30da\u30fc\u30b9\u3067\u5341\u5206\u8ffd\u3044\u4ed8\u3051\u308b\u3068\u5206\u304b\u3063\u305f\u3002<\/p>\n<h2 id=\"\u30af\u30a8\u30ea\u65b9\u6cd5\">\u30af\u30a8\u30ea\u65b9\u6cd5<\/h2>\n<p>iceberg\u3092\u5229\u7528\u3057\u3064\u3064\u73fe\u72b6\u306ecassandra\u306b\u5bfe\u3059\u308b\u30af\u30a8\u30ea\u3068\u540c\u69d8\u306b\u6570\u5206\u4ee5\u5185\u306e\u30ea\u30fc\u30c9\u30bf\u30a4\u30e0\u3067\u30c7\u30fc\u30bf\u3092\u5229\u7528\u53ef\u80fd\u306b\u3059\u308b\u305f\u3081\u306b\u3001Trino\u306eKafka connector\u3068iceberg connector\u3092UNION\u3057\u305fview\u3092\u69cb\u6210\u3057\u3001\u305d\u306eview\u306b\u5bfe\u3057\u3066\u30af\u30a8\u30ea\u3092\u884c\u3046\u3053\u3068\u3067\u6700\u65b0\u306e\u30c7\u30fc\u30bf\u3060\u3051\u3092Kafka\u304b\u3089\u53d6\u5f97\u53ef\u80fd\u306b\u306a\u308b\u3002<br \/>\n\u691c\u8a3c\u5b9f\u9a13\u3067\u306f\u3001\u73fe\u5728\u6642\u523b\u304b\u308930\u5206\u4ee5\u5185\u306etimestamp\u306b\u9650\u5b9a\u3057\u3066Kafka connector\u3067\u30af\u30a8\u30ea\u3092\u884c\u3044\u305d\u308c\u4ee5\u5916\u306e\u30c7\u30fc\u30bf\u306ficeberg\u304b\u3089\u30af\u30a8\u30ea\u3092\u884c\u3046\u69d8\u306b\u3059\u308b\u3002<\/p>\n<p>Kafka\u304b\u3089\u8aad\u307f\u8fbc\u3080\u30c7\u30fc\u30bf\u91cf\u306f\u76f4\u8fd130\u5206\u306b\u9650\u5b9a\u3059\u308b\u3068\u3001\u7d042.7GB\u30672700\u4e07\u30ec\u30b3\u30fc\u30c9\u306b\u76f8\u5f53\u3059\u308b\u3002<\/p>\n<p>Kafka\u4e0a\u306e\u30c7\u30fc\u30bf\u306ficeberg\u3068\u540c\u69d8\u306e\u30d1\u30fc\u30b7\u30e7\u30cb\u30f3\u30b0\u306f\u884c\u3048\u306a\u3044\u304c\u3001\u76f4\u8fd130\u5206\u7a0b\u5ea6\u306e\u30c7\u30fc\u30bf\u91cf\u3067\u3042\u308c\u3070\u7121\u99c4\u3092\u627f\u77e5\u306e\u4e0a\u3067\u30ec\u30b3\u30fc\u30c9\u3092\u8aad\u3093\u3067\u3082\u5927\u3057\u305f\u8ca0\u8377\u306b\u306a\u3089\u306a\u3044\u3053\u3068\u304c\u5206\u304b\u3063\u305f\u3002<\/p>\n<p>10\u53f0\u524d\u5f8c\u306etrino<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u304c\u3042\u308c\u3070\u6570\u79d2\u3067\u8aad\u3080\u3053\u3068\u304c\u51fa\u6765\u308b\u3057Kafka<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u306b\u304b\u304b\u308b\u8ca0\u8377\u3082\u8a31\u5bb9\u7bc4\u56f2\u306e\u5c0f\u3055\u3044\u3082\u306e\u3060\u3063\u305f\u3002<\/p>\n<p>iceberg\u306ecatalog\u3092Glue Catalog\u306b\u8a2d\u5b9a\u3057\u3066\u69cb\u6210\u3057\u3066\u3044\u305f\u5834\u5408\u3001iceberg\u30c6\u30fc\u30d6\u30eb\u304c\u6240\u5c5e\u3057\u3066\u3044\u308bcatalog\u53ca\u3073database\u306b\u5bfe\u3057\u3066view\u3092\u751f\u6210\u3059\u308b\u3068Glue Data Catalog\u5074\u306b\u6c38\u7d9a\u5316\u3055\u308c\u308b\u3002\u305d\u306e\u305f\u3081<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u306e\u505c\u6b62\u3084\u5165\u308c\u66ff\u3048\u3092\u4f34\u3063\u3066\u3082view\u3092\u518d\u4f5c\u6210\u3059\u308b\u624b\u9593\u306f\u304b\u304b\u3089\u306a\u3044\u3053\u3068\u3082\u78ba\u8a8d\u3067\u304d\u305f\u3002<\/p>\n<p>\u5c06\u6765\u7684\u306b\u306f<a target=\"_blank\" href=\"https:\/\/fluss.apache.org\/\">Apache Fluss<\/a>\u3092\u6d3b\u7528\u3067\u304d\u305f\u308a\u3059\u308b\u3068\u52b9\u7387\u304c\u826f\u3055\u305d\u3046\u3068\u8003\u3048\u3066\u3044\u308b\u304c\u3001\u73fe\u72b6\u3067\u306fTrino\u304b\u3089\u306e\u30af\u30a8\u30ea\u304c\u30b5\u30dd\u30fc\u30c8\u3055\u308c\u3066\u3044\u306a\u3044\u3057\u30d0\u30c3\u30af\u30a8\u30f3\u30c9\u306eiceberg\u30b5\u30dd\u30fc\u30c8\u3082\u8a08\u753b\u4e2d\u3068\u3044\u3046\u611f\u3058\u306a\u306e\u3067\u3001\u4eca\u5f8c\u306e\u5c55\u958b\u306b\u6ce8\u76ee\u3057\u3066\u3044\u304d\u305f\u3044\u3002<\/p>\n<p>Flink\u306b\u3088\u308b\u66f8\u304d\u8fbc\u307f\u306f\u52d5\u304d\u304c\u30b7\u30f3\u30d7\u30eb\u306a\u306e\u3067\u52d5\u4f5c\u81ea\u4f53\u306f\u975e\u5e38\u306b\u8efd\u3044\u3002\u691c\u8a3c\u306b\u5229\u7528\u3057\u305f\u30c6\u30fc\u30d6\u30eb\u3060\u3051\u306a\u30891\u53f0\u3067\u5341\u5206\u65e5\u4e2d\u306e\u30d4\u30fc\u30af<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%C8%A5%E9%A5%D5%A5%A3%A5%C3%A5%AF\">\u30c8\u30e9\u30d5\u30a3\u30c3\u30af<\/a>\u306b\u3082\u8010\u3048\u3089\u308c\u308b\u3002\u5bfe\u8c61\u306e\u30c6\u30fc\u30d6\u30eb\u304c\u5897\u3048\u3066\u3082\u554f\u984c\u7121\u3044\u3060\u308d\u3046\u3002<\/p>\n<p>15\u5206\u5358\u4f4d\u306eupsert\u66f8\u304d\u8fbc\u307f\u3092\u7d99\u7d9a\u3057\u3066\u3044\u308b\u72b6\u614b\u3067\u3001compaction\u306e\u5b9f\u884c\u304c\u9045\u308c\u308b\u3068\u5272\u3068\u9855\u8457\u306b\u30d1\u30d5\u30a9\u30fc\u30de\u30f3\u30b9\u306b\u5f71\u97ff\u3092\u4e0e\u3048\u308b\u3053\u3068\u304c\u5206\u304b\u3063\u305f\u3002<br \/>\n\u305f\u3060\u300124\u6642\u9593\u306b1\u5ea6\u7a0b\u5ea6\u3067\u89e3\u6d88\u3067\u304d\u3001\u60f3\u5b9a\u3055\u308c\u308b\u6700\u60aa\u306e\u72b6\u6cc1\u3067\u73fe\u5728\u306e\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u306b\u5bfe\u5fdc\u53ef\u80fd\u306a\u7bc4\u7587\u306a\u306e\u3067\u3001\u904b\u7528\u8ca0\u8377\u3068\u3057\u3066\u306f\u5341\u5206\u306b\u8a31\u5bb9\u7bc4\u56f2\u3067\u5bfe\u51e6\u3067\u304d\u308b\u7bc4\u56f2\u3068\u8a00\u3048\u308b\u3002<\/p>\n<p>\u3057\u304b\u3057\u3001compaction\u306f\u304b\u306a\u308a\u51e6\u7406\u8ca0\u8377\u304c\u9ad8\u304f\u30e1\u30e2\u30ea\u3082CPU\u30ea\u30bd\u30fc\u30b9\u3082\u304b\u306a\u308a\u5fc5\u8981\u306b\u306a\u308b\u3002\u7279\u306b\u4eca\u306eiceberg\u3067\u666e\u53ca\u3057\u3066\u3044\u308b\u30c6\u30fc\u30d6\u30eb\u4ed5\u69d8\u306e\u30d0\u30fc\u30b8\u30e7\u30f3(v2)\u3068spark\u306e\u5b9f\u88c5\u3067\u306f\u3001\u975e\u5e38\u306b\u5927\u91cf\u306e\u5c0f\u3055\u306a\u30d5\u30a1\u30a4\u30eb(\u7279\u306bdelete\u30d5\u30a1\u30a4\u30eb)\u304c\u5b58\u5728\u3059\u308b\u3068\u3001compaction\u306b\u81a8\u5927\u306a\u30e1\u30e2\u30ea\u304c\u5fc5\u8981\u306b\u306a\u308b\u305f\u3081\u3001compaction\u304c\u9577\u671f\u306b\u6e21\u3063\u3066\u5b9f\u884c\u3055\u308c\u306a\u3044\u3068\u30c6\u30fc\u30d6\u30eb\u306e\u30e1\u30f3\u30c6\u30ca\u30f3\u30b9\u81ea\u4f53\u304c\u56f0\u96e3\u306b\u306a\u308b\u53ef\u80fd\u6027\u304c\u3042\u3063\u3066\u5371\u967a\u3002<\/p>\n<p>\u3068\u3044\u3046\u8a33\u3067compaction\u306e\u6240\u8981\u6642\u9593\u3092\u305d\u308c\u306a\u308a\u306b\u6291\u3048\u3066\u5b89\u5b9a\u3057\u3066\u5b9f\u884c\u3067\u304d\u308b\u69d8\u306b\u3059\u308b\u306b\u306f\u5272\u3068\u5927\u304d\u3081\u306espark<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u304c\u3044\u308b\u3002\u30c6\u30fc\u30d6\u30eb\u3092\u5897\u3084\u3057\u3066\u3044\u3051\u3070\u30ef\u30f3\u30bf\u30a4\u30e0\u306e\u5b9f\u884c\u3067\u3082\u591a\u5c11\u306e\u30b3\u30b9\u30c8\u306f\u5fc5\u8981\u306b\u306a\u308b\u3060\u308d\u3046\u3002<br \/>\n\u30c6\u30fc\u30d6\u30eb\u306etarget size\u3084\u30d0\u30b1\u30c3\u30c6\u30a3\u30f3\u30b0\u306e\u6570\u3092\u8abf\u6574\u3059\u308b\u3053\u3068\u3067\u3001compaction\u8ca0\u8377\u3068\u30af\u30a8\u30ea\u8ca0\u8377\u306e\u30d0\u30e9\u30f3\u30b9\u3092\u53d6\u308b\u3053\u3068\u306f\u51fa\u6765\u308b\u306e\u3067\u30c6\u30fc\u30d6\u30eb\u8a2d\u8a08\u6642\u306b\u3057\u3063\u304b\u308a\u8003\u3048\u308b\u5fc5\u8981\u304c\u3042\u308a\u305d\u3046\u3002<\/p>\n<p>\u30af\u30a8\u30ea\u306b\u3064\u3044\u3066\u306f\u3001Kafka Connector\u3068\u306e\u7d44\u307f\u5408\u308f\u305b\u304c\u73fe\u5b9f\u7684\u306b\u5229\u7528\u3067\u304d\u308b\u3050\u3089\u3044\u306e\u8ca0\u8377\u3068\u30ec\u30a4\u30c6\u30f3\u30b7\u3067\u7d50\u679c\u3092\u8fd4\u305b\u308b\u3053\u3068\u304c\u5206\u304b\u3063\u305f\u3002Glue Data Catalog\u306bview\u3092\u4fdd\u5b58\u3057\u3066\u304a\u3051\u308b\u3053\u3068\u304c\u5206\u304b\u3063\u305f\u70b9\u3082\u5927\u304d\u3044\u3002\u72ec\u7acb\u3057\u305f\u30de\u30cd\u30fc\u30b8\u30c9\u30b5\u30fc\u30d3\u30b9\u306b\u6c38\u7d9a\u5316\u3067\u304d\u308b\u306e\u3067<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u306e\u518d\u69cb\u6210\u3068view\u306e\u518d\u69cb\u6210\u3092\u5206\u96e2\u3067\u304d\u308b\u3002\u305f\u3060\u3057\u3001\u540c\u6642\u306b\u8907\u6570\u306e\u30af\u30a8\u30ea\u304c\u767a\u751f\u3057\u305f\u6642\u306eKafka Broker\u3078\u306e\u8ca0\u8377\u306f\u5225\u9014\u691c\u8a3c\u3057\u305f\u65b9\u304c\u826f\u3044\u304b\u3082\u3057\u308c\u306a\u3044\u3002<\/p>\n<p>\u4eca\u306eRepro\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%E6%A1%BC%A5%B9%A5%B1%A1%BC%A5%B9\">\u30e6\u30fc\u30b9\u30b1\u30fc\u30b9<\/a>\u3068\u30ef\u30fc\u30af\u30ed\u30fc\u30c9\u3092\u60f3\u5b9a\u3057\u305f\u7dcf\u5408\u7684\u306a\u8a55\u4fa1\u3068\u3057\u3066\u306f\u3001\u3044\u304f\u3064\u304b\u306e\u61f8\u5ff5\u4e8b\u9805\u306f\u3042\u308b\u304c\u5341\u5206production\u306b\u6295\u5165\u53ef\u80fd\u3067\u3042\u308b\u3068\u5224\u65ad\u3059\u308b\u3002\u66f8\u304d\u8fbc\u307f\u52d5\u4f5c\u306e\u5b89\u5b9a\u6027\u81ea\u4f53\u306f\u9ad8\u304f\u3001\u660e\u3089\u304b\u306b\u5bfe\u51e6\u4e0d\u53ef\u80fd\u306a\u8ab2\u984c\u306f\u5b58\u5728\u3057\u306a\u304b\u3063\u305f\u3002<\/p>\n<p>\u5b9f\u969b\u306bproduction\u3067\u904b\u7528\u3059\u308b\u305f\u3081\u306b\u306f\u3001\u904b\u7528\u306e\u5b89\u5b9a\u5316\u306e\u305f\u3081\u306b\u3044\u304f\u3064\u304b\u5fc5\u8981\u306a\u3082\u306e\u304c\u3042\u308b\u3002<\/p>\n<h2 id=\"\u30c6\u30fc\u30d6\u30eb\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0\">\u30c6\u30fc\u30d6\u30eb\u30c1\u30e5\u30fc\u30cb\u30f3\u30b0<\/h2>\n<p>\u691c\u8a3c\u3067\u5f97\u305f\u60c5\u5831\u3092\u8e0f\u307e\u3048\u3066\u3001\u30c6\u30fc\u30d6\u30eb\u306e\u8a2d\u5b9a\u5024(\u7279\u306b\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30cb\u30f3\u30b0)\u306b\u3064\u3044\u3066\u3088\u308a\u3088\u3044\u30d0\u30e9\u30f3\u30b9\u3092\u8a2d\u5b9a\u3057\u305f\u3044\u3002<\/p>\n<h2 id=\"\u30e2\u30cb\u30bf\u30ea\u30f3\u30b0\u76e3\u8996\">\u30e2\u30cb\u30bf\u30ea\u30f3\u30b0\u30fb\u76e3\u8996<\/h2>\n<ul>\n<li>flink\u304c\u3061\u3083\u3093\u3068\u52d5\u4f5c\u3057\u3066\u3044\u308b\u3053\u3068\u3092\u76e3\u8996\u3057\u3001\u30de\u30b7\u30f3\u30ea\u30bd\u30fc\u30b9\u3068\u66f8\u304d\u8fbc\u307f\u30da\u30fc\u30b9\u306e\u30e1\u30c8\u30ea\u30c3\u30af\u3092\u53d6\u5f97\u3067\u304d\u308b\u69d8\u306b\u3059\u308b\u3002\n<ul>\n<li>Flink\u306ereporter\u304b\u3089datadog\u306b\u76f4\u3067\u30e1\u30c8\u30ea\u30c3\u30af\u9001\u308c\u308b\u69d8\u306b\u3057\u305f\u3044<\/li>\n<\/ul>\n<\/li>\n<li>compaction\u3092\u5b9a\u671f\u7684\u306b\u5b9f\u884c\u3059\u308b\u30b9\u30b1\u30b8\u30e5\u30fc\u30e9\u30fc\u3002\u6050\u3089\u304fEventBridge\u3068StepFunction\u3067\u826f\u3044\u306f\u305a\u3002\n<ul>\n<li>\u5b9f\u884c\u306e\u4ed5\u7d44\u307f\u306e\u4e2d\u3067compaction\u306e\u5931\u6557\u6642\u306b\u30a2\u30e9\u30fc\u30c8\u3092\u4e0a\u3052\u3089\u308c\u308b\u69d8\u306b\u3059\u308b\u4ed5\u7d44\u307f\u304c\u5fc5\u8981\u3002<\/li>\n<\/ul>\n<\/li>\n<li>iceberg\u306e\u30c6\u30fc\u30d6\u30eb<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%E1%A5%BF%A5%C7%A1%BC%A5%BF\">\u30e1\u30bf\u30c7\u30fc\u30bf<\/a>\u306b\u3088\u308b\u30e2\u30cb\u30bf\u30ea\u30f3\u30b0\u3002<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u3054\u3068\u306e\u30d5\u30a1\u30a4\u30eb\u6570\u3084\u30d5\u30a1\u30a4\u30eb\u30b5\u30a4\u30ba\u5408\u8a08\u3001\u5229\u7528\u53ef\u80fd\u306a\u30b9\u30ca\u30c3\u30d7\u30b7\u30e7\u30c3\u30c8\u306e\u6570\u306a\u3069<\/li>\n<\/ul>\n<p>\u4e0a\u8a18\u304c\u3001\u4eca\u56deIceberg\u306e\u6280\u8853\u691c\u8a3c\u3068\u3057\u3066\u884c\u306a\u3063\u305f\u5185\u5bb9\u3068\u8a55\u4fa1\u7d50\u679c\u3067\u3059\u3002<br \/>\n\u30a2\u30a6\u30c8\u30d7\u30c3\u30c8\u3068\u3057\u3066\u3042\u308b\u7a0b\u5ea6\u30b7\u30f3\u30d7\u30eb\u306b\u307e\u3068\u3081\u3066\u3042\u308a\u307e\u3059\u304c\u3001\u5b9f\u969b\u306f<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%AF%A5%E9%A5%B9%A5%BF\">\u30af\u30e9\u30b9\u30bf<\/a>\u69cb\u7bc9\u304b\u3089\u5b9f\u969b\u306b\u66f8\u304d\u8fbc\u307f\u3092\u884c\u3063\u3066\u52d5\u4f5c\u78ba\u8a8d\u3092\u3059\u308b\u307e\u3067\u306b\u3001S3\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D0%A5%B1%A5%C3%A5%C8\">\u30d0\u30b1\u30c3\u30c8<\/a>\u306e\u6574\u5099\u3084IAM\u306e\u6a29\u9650\u306e\u4fee\u6b63\u3067terraform\u3092\u76f4\u3057\u305f\u308a\u3001\u5b9f\u884c\u4e2d\u306b<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/AWS\">AWS<\/a>\u306e\u30b3\u30b9\u30c8\u306e\u5909\u52d5\u3092\u30e2\u30cb\u30bf\u3057\u305f\u308a\u3001production\u306e<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%DF%A5%C9%A5%EB%A5%A6%A5%A7%A5%A2\">\u30df\u30c9\u30eb\u30a6\u30a7\u30a2<\/a>\u3092\u3042\u308b\u7a0b\u5ea6\u5229\u7528\u3059\u308b\u306e\u3067\u8ca0\u8377\u3092\u304b\u3051\u3066\u3082\u5927\u4e08\u592b\u306a\u6642\u9593\u306b\u5b9f\u9a13\u3057\u305f\u308a\u3068\u3001\u305d\u308c\u306a\u308a\u306b\u6642\u9593\u304c\u304b\u304b\u3063\u3066\u3044\u307e\u3059\u3002<\/p>\n<p>\u7279\u306bcompaction\u306e\u5b9f\u884c\u306b\u306f\u7d50\u69cb\u306a\u6642\u9593\u3068\u30ea\u30bd\u30fc\u30b9\u304c\u5fc5\u8981\u3060\u3063\u305f\u306e\u3067\u3001\u3053\u3053\u304c\u5b89\u5b9a\u3057\u3066\u52d5\u4f5c\u3059\u308b\u72b6\u6cc1\u306b\u6301\u3063\u3066\u3044\u304f\u306e\u306f\u7d50\u69cb\u82e6\u52b4\u3057\u307e\u3057\u305f\u3002<br \/>\n\u3084\u306f\u308a\u3001\u3053\u306e\u624b\u306eOpen Table Format\u3092Update\u3042\u308a\u304d\u3067\u904b\u7528\u3057\u3088\u3046\u3068\u3057\u305f\u3089\u3001\u91cd\u8981\u306b\u306a\u308b\u306e\u306f\u30c6\u30fc\u30d6\u30eb\u306ecompaction\u3068<a target=\"_blank\" class=\"keyword\" href=\"https:\/\/d.hatena.ne.jp\/keyword\/%A5%D1%A1%BC%A5%C6%A5%A3%A5%B7%A5%E7%A5%F3\">\u30d1\u30fc\u30c6\u30a3\u30b7\u30e7\u30f3<\/a>\u6570\u306e\u30d0\u30e9\u30f3\u30b9\u3060\u306a\u3068\u3044\u3046\u3053\u3068\u3092\u6539\u3081\u3066\u5b9f\u611f\u3067\u304d\u307e\u3057\u305f\u3002<\/p>\n<p>\u3053\u306e\u8a18\u4e8b\u304c\u3001Iceberg\u3092\u5229\u7528\u3057\u3088\u3046\u3068\u3057\u3066\u3044\u308b\u8ab0\u304b\u306e\u53c2\u8003\u306b\u306a\u308c\u3070\u5e78\u3044\u3067\u3059\u3002<\/p>\n<\/div>\n<p><script>(function(d, s, id) {\n  var js, fjs = d.getElementsByTagName(s)[0];\n  if (d.getElementById(id)) return;\n  js = d.createElement(s); js.id = id;\n  js.src = \"\/\/connect.facebook.net\/ja_JP\/sdk.js#xfbml=1&appId=719729204785177&version=v17.0\";\n  fjs.parentNode.insertBefore(js, fjs);\n}(document, 'script', 'facebook-jssdk'));<\/script><br \/>\n<br \/>\n<br \/><a href=\"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646\">\u5143\u306e\u8a18\u4e8b\u3092\u78ba\u8a8d\u3059\u308b <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Repro\u3067\u30c1\u30fc\u30d5\u30a2\u30fc\u30ad\u30c6\u30af\u30c8\u3068\u3057\u3066\u4ed5\u4e8b\u3092\u3057\u3066\u3044\u308b\u6a4b\u7acb(joker1007)\u3067\u3059\u3002 \u4eca\u56de\u3001\u672c\u756a\u74b0\u5883\u3078\u306e\u5c0e\u5165\u3092\u8996\u91ce\u306b\u5165\u308c\u3066Apache Iceberg\u3092\u5b9f\u969b\u306b\u696d\u52d9\u30c7\u30fc\u30bf\u3067\u691c\u8a3c\u3057\u307e\u3057\u305f\u3002 \u3053\u306e\u8a18\u4e8b\u3067\u306f\u305d\u306e\u691c\u8a3c\u5185\u5bb9\u3068\u7d50\u679c\u306b\u3064\u3044 [&hellip;]","protected":false},"author":1,"featured_media":23543,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-23542","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-hatena-blog"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.6 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c - Repro Tech Blog - \u30dd\u30b1\u30b3\u30f3<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646\" \/>\n<meta property=\"og:locale\" content=\"ja_JP\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c - Repro Tech Blog - \u30dd\u30b1\u30b3\u30f3\" \/>\n<meta property=\"og:description\" content=\"Repro\u3067\u30c1\u30fc\u30d5\u30a2\u30fc\u30ad\u30c6\u30af\u30c8\u3068\u3057\u3066\u4ed5\u4e8b\u3092\u3057\u3066\u3044\u308b\u6a4b\u7acb(joker1007)\u3067\u3059\u3002 \u4eca\u56de\u3001\u672c\u756a\u74b0\u5883\u3078\u306e\u5c0e\u5165\u3092\u8996\u91ce\u306b\u5165\u308c\u3066Apache Iceberg\u3092\u5b9f\u969b\u306b\u696d\u52d9\u30c7\u30fc\u30bf\u3067\u691c\u8a3c\u3057\u307e\u3057\u305f\u3002 \u3053\u306e\u8a18\u4e8b\u3067\u306f\u305d\u306e\u691c\u8a3c\u5185\u5bb9\u3068\u7d50\u679c\u306b\u3064\u3044 [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646\" \/>\n<meta property=\"og:site_name\" content=\"\u30dd\u30b1\u30b3\u30f3\" \/>\n<meta property=\"article:published_time\" content=\"2025-11-13T18:35:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/pokecon.jp\/job\/wp-content\/uploads\/2025\/11\/1763018206.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"info@pokecon.jp\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"\u57f7\u7b46\u8005\" \/>\n\t<meta name=\"twitter:data1\" content=\"info@pokecon.jp\" \/>\n\t<meta name=\"twitter:label2\" content=\"\u63a8\u5b9a\u8aad\u307f\u53d6\u308a\u6642\u9593\" \/>\n\t<meta name=\"twitter:data2\" content=\"5\u5206\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/23542\\\/\"},\"author\":{\"name\":\"info@pokecon.jp\",\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/#\\\/schema\\\/person\\\/16c9f07b1ba984d165d9aee259bda997\"},\"headline\":\"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c &#8211; Repro Tech Blog\",\"datePublished\":\"2025-11-13T18:35:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/23542\\\/\"},\"wordCount\":328,\"image\":{\"@id\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/1763018206.png\",\"articleSection\":[\"\u306f\u3066\u306a\u30d6\u30ed\u30b0\"],\"inLanguage\":\"ja\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/23542\\\/\",\"url\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646\",\"name\":\"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c - Repro Tech Blog - \u30dd\u30b1\u30b3\u30f3\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/1763018206.png\",\"datePublished\":\"2025-11-13T18:35:41+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/#\\\/schema\\\/person\\\/16c9f07b1ba984d165d9aee259bda997\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646#breadcrumb\"},\"inLanguage\":\"ja\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"ja\",\"@id\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646#primaryimage\",\"url\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/1763018206.png\",\"contentUrl\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/wp-content\\\/uploads\\\/2025\\\/11\\\/1763018206.png\",\"width\":1200,\"height\":630},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/tech.repro.io\\\/entry\\\/2025\\\/11\\\/13\\\/161646#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"\u30db\u30fc\u30e0\",\"item\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c &#8211; Repro Tech Blog\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/#website\",\"url\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/\",\"name\":\"\u30dd\u30b1\u30b3\u30f3\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ja\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/#\\\/schema\\\/person\\\/16c9f07b1ba984d165d9aee259bda997\",\"name\":\"info@pokecon.jp\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"ja\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/2b0549cd9f7907c092ca5fbb283baf72337f235726e4b46fa39ec0b701ac2fe2?s=96&d=wavatar&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/2b0549cd9f7907c092ca5fbb283baf72337f235726e4b46fa39ec0b701ac2fe2?s=96&d=wavatar&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/2b0549cd9f7907c092ca5fbb283baf72337f235726e4b46fa39ec0b701ac2fe2?s=96&d=wavatar&r=g\",\"caption\":\"info@pokecon.jp\"},\"url\":\"https:\\\/\\\/pokecon.jp\\\/job\\\/author\\\/infopokecon-jp\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c - Repro Tech Blog - \u30dd\u30b1\u30b3\u30f3","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646","og_locale":"ja_JP","og_type":"article","og_title":"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c - Repro Tech Blog - \u30dd\u30b1\u30b3\u30f3","og_description":"Repro\u3067\u30c1\u30fc\u30d5\u30a2\u30fc\u30ad\u30c6\u30af\u30c8\u3068\u3057\u3066\u4ed5\u4e8b\u3092\u3057\u3066\u3044\u308b\u6a4b\u7acb(joker1007)\u3067\u3059\u3002 \u4eca\u56de\u3001\u672c\u756a\u74b0\u5883\u3078\u306e\u5c0e\u5165\u3092\u8996\u91ce\u306b\u5165\u308c\u3066Apache Iceberg\u3092\u5b9f\u969b\u306b\u696d\u52d9\u30c7\u30fc\u30bf\u3067\u691c\u8a3c\u3057\u307e\u3057\u305f\u3002 \u3053\u306e\u8a18\u4e8b\u3067\u306f\u305d\u306e\u691c\u8a3c\u5185\u5bb9\u3068\u7d50\u679c\u306b\u3064\u3044 [&hellip;]","og_url":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646","og_site_name":"\u30dd\u30b1\u30b3\u30f3","article_published_time":"2025-11-13T18:35:41+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/pokecon.jp\/job\/wp-content\/uploads\/2025\/11\/1763018206.png","type":"image\/png"}],"author":"info@pokecon.jp","twitter_card":"summary_large_image","twitter_misc":{"\u57f7\u7b46\u8005":"info@pokecon.jp","\u63a8\u5b9a\u8aad\u307f\u53d6\u308a\u6642\u9593":"5\u5206"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646#article","isPartOf":{"@id":"https:\/\/pokecon.jp\/job\/23542\/"},"author":{"name":"info@pokecon.jp","@id":"https:\/\/pokecon.jp\/job\/#\/schema\/person\/16c9f07b1ba984d165d9aee259bda997"},"headline":"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c &#8211; Repro Tech Blog","datePublished":"2025-11-13T18:35:41+00:00","mainEntityOfPage":{"@id":"https:\/\/pokecon.jp\/job\/23542\/"},"wordCount":328,"image":{"@id":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646#primaryimage"},"thumbnailUrl":"https:\/\/pokecon.jp\/job\/wp-content\/uploads\/2025\/11\/1763018206.png","articleSection":["\u306f\u3066\u306a\u30d6\u30ed\u30b0"],"inLanguage":"ja"},{"@type":"WebPage","@id":"https:\/\/pokecon.jp\/job\/23542\/","url":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646","name":"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c - Repro Tech Blog - \u30dd\u30b1\u30b3\u30f3","isPartOf":{"@id":"https:\/\/pokecon.jp\/job\/#website"},"primaryImageOfPage":{"@id":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646#primaryimage"},"image":{"@id":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646#primaryimage"},"thumbnailUrl":"https:\/\/pokecon.jp\/job\/wp-content\/uploads\/2025\/11\/1763018206.png","datePublished":"2025-11-13T18:35:41+00:00","author":{"@id":"https:\/\/pokecon.jp\/job\/#\/schema\/person\/16c9f07b1ba984d165d9aee259bda997"},"breadcrumb":{"@id":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646#breadcrumb"},"inLanguage":"ja","potentialAction":[{"@type":"ReadAction","target":["https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646"]}]},{"@type":"ImageObject","inLanguage":"ja","@id":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646#primaryimage","url":"https:\/\/pokecon.jp\/job\/wp-content\/uploads\/2025\/11\/1763018206.png","contentUrl":"https:\/\/pokecon.jp\/job\/wp-content\/uploads\/2025\/11\/1763018206.png","width":1200,"height":630},{"@type":"BreadcrumbList","@id":"https:\/\/tech.repro.io\/entry\/2025\/11\/13\/161646#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"\u30db\u30fc\u30e0","item":"https:\/\/pokecon.jp\/job\/"},{"@type":"ListItem","position":2,"name":"Repro\u3067\u306eApache Iceberg\u306e\u6280\u8853\u691c\u8a3c\u7d50\u679c &#8211; Repro Tech Blog"}]},{"@type":"WebSite","@id":"https:\/\/pokecon.jp\/job\/#website","url":"https:\/\/pokecon.jp\/job\/","name":"\u30dd\u30b1\u30b3\u30f3","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/pokecon.jp\/job\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ja"},{"@type":"Person","@id":"https:\/\/pokecon.jp\/job\/#\/schema\/person\/16c9f07b1ba984d165d9aee259bda997","name":"info@pokecon.jp","image":{"@type":"ImageObject","inLanguage":"ja","@id":"https:\/\/secure.gravatar.com\/avatar\/2b0549cd9f7907c092ca5fbb283baf72337f235726e4b46fa39ec0b701ac2fe2?s=96&d=wavatar&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/2b0549cd9f7907c092ca5fbb283baf72337f235726e4b46fa39ec0b701ac2fe2?s=96&d=wavatar&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/2b0549cd9f7907c092ca5fbb283baf72337f235726e4b46fa39ec0b701ac2fe2?s=96&d=wavatar&r=g","caption":"info@pokecon.jp"},"url":"https:\/\/pokecon.jp\/job\/author\/infopokecon-jp\/"}]}},"_links":{"self":[{"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/posts\/23542","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/comments?post=23542"}],"version-history":[{"count":1,"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/posts\/23542\/revisions"}],"predecessor-version":[{"id":23544,"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/posts\/23542\/revisions\/23544"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/media\/23543"}],"wp:attachment":[{"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/media?parent=23542"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/categories?post=23542"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pokecon.jp\/job\/wp-json\/wp\/v2\/tags?post=23542"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}