备案控制台
开发者社区 开发与运维 文章 正文

Spark实践-日志查询

2016-04-29 1150
版权
版权声明：
本文内容由阿里云实名注册用户自发贡献，版权归原作者所有，阿里云开发者社区不拥有其著作权，亦不承担相应法律责任。具体规则请查看《 阿里云开发者社区用户服务协议》和 《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容，填写 侵权投诉表单进行举报，一经查实，本社区将立刻删除涉嫌侵权内容。
简介： 环境 win 7 jdk 1.7.0_79 (Oracle Corporation) scala version 2.10.5 spark 1.
  1. 环境
    win 7
    jdk 1.7.0_79 (Oracle Corporation)
    scala version 2.10.5
    spark 1.6.1
    详细配置：
    Spark Properties
spark.app.id    local-1461891171126
spark.app.name  JavaLogQuery
spark.driver.host   10.170.26.123
spark.driver.port   34998
spark.executor.id   driver
spark.externalBlockStore.folderName spark-5242ec5b-3653-42e4-9ba2-da3ef515a1d5
spark.master    local[1]
spark.scheduler.mode    FIFO
  1. 任务

完成对如下日志的查询：

 "10.10.10.10 - \"FRED\" [18/Jan/2013:17:56:07 +1100] \"GET http://images.com/2013/Generic.jpg " +
                    "HTTP/1.1\" 304 315 \"http://referall.com/\" \"Mozilla/4.0 (compatible; MSIE 7.0; " +
                    "Windows NT 5.1; GTB7.4; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; " +
                    ".NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR " +
                    "3.5.30729; Release=ARP)\" \"UD-1\" - \"image/jpeg\" \"whatever\" 0.350 \"-\" - \"\" 265 923 934 \"\" " +
                    "62.24.11.25 images.com 1358492167 - Whatup",
                    "10.10.10.10 - \"FRED\" [18/Jan/2013:18:02:37 +1100] \"GET http://images.com/2013/Generic.jpg " +
                            "HTTP/1.1\" 304 306 \"http:/referall.com\" \"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; " +
                            "GTB7.4; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR " +
                            "3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR  " +
                            "3.5.30729; Release=ARP)\" \"UD-1\" - \"image/jpeg\" \"whatever\" 0.352 \"-\" - \"\" 256 977 988 \"\" " +
            "0 73.23.2.15 images.com 1358492557 - Whatup"

思路：
1.利用正则表达式提取出日志特征，然后map在分片后的RDD上。

JavaPairRDD<Tuple3<String, String, String>, Stats> extracted

2.执行reducebykey,merge相同的Stats


package org.apache.spark.examples;

import com.google.common.collect.Lists;
import scala.Tuple2;
import scala.Tuple3;

import org.apache.commons.logging.impl.Log4JLogger;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function2;
import org.apache.spark.api.java.function.PairFunction;

import java.io.Serializable;
import java.util.Collections;
import java.util.List;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * 日志查询
 * @author jinhang
 *
 */
public final class JavaLogQuery {
    //模拟日志 exampleApacheLogs
    public static final List<String> exampleApacheLogs = Lists.newArrayList(
            "10.10.10.10 - \"FRED\" [18/Jan/2013:17:56:07 +1100] \"GET http://images.com/2013/Generic.jpg " +
                    "HTTP/1.1\" 304 315 \"http://referall.com/\" \"Mozilla/4.0 (compatible; MSIE 7.0; " +
                    "Windows NT 5.1; GTB7.4; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; " +
                    ".NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR " +
                    "3.5.30729; Release=ARP)\" \"UD-1\" - \"image/jpeg\" \"whatever\" 0.350 \"-\" - \"\" 265 923 934 \"\" " +
                    "62.24.11.25 images.com 1358492167 - Whatup",
                    "10.10.10.10 - \"FRED\" [18/Jan/2013:18:02:37 +1100] \"GET http://images.com/2013/Generic.jpg " +
                            "HTTP/1.1\" 304 306 \"http:/referall.com\" \"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; " +
                            "GTB7.4; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR " +
                            "3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR  " +
                            "3.5.30729; Release=ARP)\" \"UD-1\" - \"image/jpeg\" \"whatever\" 0.352 \"-\" - \"\" 256 977 988 \"\" " +
            "0 73.23.2.15 images.com 1358492557 - Whatup");

    public static final Pattern apacheLogRegex = Pattern.compile(
            "^([\\d.]+) (\\S+) (\\S+) \\[([\\w\\d:/]+\\s[+\\-]\\d{4})\\] \"(.+?)\" (\\d{3}) ([\\d\\-]+) \"([^\"]+)\" \"([^\"]+)\".*");

    public static class Stats implements Serializable {

        private final int count;
        private final int numBytes;

        public Stats(int count, int numBytes) {
            this.count = count;
            this.numBytes = numBytes;
        }
        public Stats merge(Stats other) {
            return new Stats(count + other.count, numBytes + other.numBytes);
        }

        public String toString() {
            return String.format("bytes=%s\tn=%s", numBytes, count);
        }
    }

    public static Tuple3<String, String, String> extractKey(String line) {
        Matcher m = apacheLogRegex.matcher(line);
        if (m.find()) {
            String ip = m.group(1);
            String user = m.group(3);
            String query = m.group(5);

            if (!user.equalsIgnoreCase("-")) {
                return new Tuple3<String, String, String>(ip, user, query);
            }
        }
        return new Tuple3<String, String, String>(null, null, null);
    }

    public static Stats extractStats(String line) {
        Matcher m = apacheLogRegex.matcher(line);
        if (m.find()) {
            int bytes = Integer.parseInt(m.group(7));
            return new Stats(1, bytes);
        } else {
            return new Stats(1, 0);
        }
    }

    public static void main(String[] args) {
        Logger.getLogger(JavaLogQuery.class).setLevel(Level.FATAL);
        SparkConf sparkConf = new SparkConf().setAppName("JavaLogQuery").setMaster("local[1]");
        JavaSparkContext jsc = new JavaSparkContext(sparkConf);

        JavaRDD<String> dataSet = (args.length == 1) ? jsc.textFile(args[0]) : jsc.parallelize(exampleApacheLogs);

        JavaPairRDD<Tuple3<String, String, String>, Stats> extracted = dataSet.mapToPair(new PairFunction<String, Tuple3<String, String, String>, Stats>() {
            @Override
            public Tuple2<Tuple3<String, String, String>, Stats> call(String s) {
                return new Tuple2<Tuple3<String, String, String>, Stats>(extractKey(s), extractStats(s));
            }
        });

        JavaPairRDD<Tuple3<String, String, String>, Stats> counts = extracted.reduceByKey(new Function2<Stats, Stats, Stats>() {
            @Override
            public Stats call(Stats stats, Stats stats2) {
                return stats.merge(stats2);
            }
        });

        List<Tuple2<Tuple3<String, String, String>, Stats>> output = counts.collect();
        //遍历结果
        for (Tuple2<?,?> t : output) {
            System.out.println(t._1() + "\t" + t._2());
        }
        jsc.stop();
    }
}

这里写图片描述

分析下执行过程：

加载SLF4J

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/D:/JavaProject/spark-demo/lib/spark-assembly-1.6.1-hadoop2.0.0-mr1-cdh4.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/D:/JavaProject/spark-demo/lib/spark-examples-1.6.1-hadoop2.0.0-mr1-cdh4.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

初始化sparkcontext上下文

16/04/29 09:36:22 INFO SparkContext: Running Spark version 1.6.1
//-Djava.library.path=$HADOOP_HOME/lib/native/Linux-amd64-64/*.jar可以解决
16/04/29 09:36:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/04/29 09:36:23 INFO SecurityManager: Changing view acls to: hp
16/04/29 09:36:23 INFO SecurityManager: Changing modify acls to: hp
16/04/29 09:36:23 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hp); users with modify permissions: Set(hp)
16/04/29 09:36:23 INFO Utils: Successfully started service 'sparkDriver' on port 36010.
16/04/29 09:36:23 INFO Slf4jLogger: Slf4jLogger started
16/04/29 09:36:24 INFO Remoting: Starting remoting
16/04/29 09:36:24 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.170.26.123:36023]
16/04/29 09:36:24 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 36023.
16/04/29 09:36:24 INFO SparkEnv: Registering MapOutputTracker
16/04/29 09:36:24 INFO SparkEnv: Registering BlockManagerMaster
16/04/29 09:36:24 INFO DiskBlockManager: Created local directory at C:\Users\hp\AppData\Local\Temp\blockmgr-84667505-0018-439b-9627-a4360d872118
16/04/29 09:36:24 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
16/04/29 09:36:24 INFO SparkEnv: Registering OutputCommitCoordinator
16/04/29 09:36:24 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/04/29 09:36:24 INFO SparkUI: Started SparkUI at http://10.170.26.123:4040
16/04/29 09:36:24 INFO Executor: Starting executor ID driver on host localhost
16/04/29 09:36:24 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 36030.
16/04/29 09:36:24 INFO NettyBlockTransferService: Server created on 36030
16/04/29 09:36:24 INFO BlockManagerMaster: Trying to register BlockManager
16/04/29 09:36:24 INFO BlockManagerMasterEndpoint: Registering block manager localhost:36030 with 517.4 MB RAM, BlockManagerId(driver, localhost, 36030)
16/04/29 09:36:24 INFO BlockManagerMaster: Registered BlockManager

这里写图片描述
SecurityManager
‘sparkDriver’ on port 36010
Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.170.26.123:36023]
MapOutputTracker
BlockManagerMaster
DiskBlockManager: Created local directory at C:\Users\hp\AppData\Local\Temp\blockmgr-84667505-0018-439b-9627-
OutputCommitCoordinator
Executor
org.apache.spark.network.netty.NettyBlockTransferService
这几个是几个主要过程。

开始执行job

16/04/29 10:12:31 INFO SparkContext: Starting job: collect at JavaLogQuery.java:112
16/04/29 10:12:31 INFO DAGScheduler: Registering RDD 1 (mapToPair at JavaLogQuery.java:98)
16/04/29 10:12:31 INFO DAGScheduler: Got job 0 (collect at JavaLogQuery.java:112) with 1 output partitions
16/04/29 10:12:31 INFO DAGScheduler: Final stage: ResultStage 1 (collect at JavaLogQuery.java:112)
16/04/29 10:12:31 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
16/04/29 10:12:31 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
16/04/29 10:12:31 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at mapToPair at JavaLogQuery.java:98), which has no missing parents
16/04/29 10:12:31 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
16/04/29 10:12:31 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.1 KB, free 3.1 KB)
16/04/29 10:12:31 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1897.0 B, free 5.0 KB)
16/04/29 10:12:31 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:36394 (size: 1897.0 B, free: 517.4 MB)
16/04/29 10:12:31 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1006
16/04/29 10:12:31 INFO DAGScheduler: Submitting 1 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at mapToPair at JavaLogQuery.java:98)
16/04/29 10:12:31 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/04/29 10:12:31 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 3033 bytes)
16/04/29 10:12:31 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/04/29 10:12:32 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1158 bytes result sent to driver
16/04/29 10:12:32 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 320 ms on localhost (1/1)
16/04/29 10:12:32 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
16/04/29 10:12:32 INFO DAGScheduler: ShuffleMapStage 0 (mapToPair at JavaLogQuery.java:98) finished in 0.349 s
16/04/29 10:12:32 INFO DAGScheduler: looking for newly runnable stages
16/04/29 10:12:32 INFO DAGScheduler: running: Set()
16/04/29 10:12:32 INFO DAGScheduler: waiting: Set(ResultStage 1)
16/04/29 10:12:32 INFO DAGScheduler: failed: Set()
16/04/29 10:12:32 INFO DAGScheduler: Submitting ResultStage 1 ***(ShuffledRDD[2] at reduceByKey at JavaLogQuery.java:105), which has no missing parents***
16/04/29 10:12:32 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 7.9 KB)
16/04/29 10:12:32 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1746.0 B, free 9.6 KB)
16/04/29 10:12:32 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:36394 (size: 1746.0 B, free: 517.4 MB)
16/04/29 10:12:32 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
16/04/29 10:12:32 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (ShuffledRDD[2] at reduceByKey at JavaLogQuery.java:105)
16/04/29 10:12:32 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/04/29 10:12:32 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, partition 0,NODE_LOCAL, 1894 bytes)
16/04/29 10:12:32 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
16/04/29 10:12:32 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks
16/04/29 10:12:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 16 ms
16/04/29 10:12:32 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1449 bytes result sent to driver
16/04/29 10:12:32 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 107 ms on localhost (1/1)
16/04/29 10:12:32 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
16/04/29 10:12:32 INFO DAGScheduler: ResultStage 1 (collect at JavaLogQuery.java:112) finished in 0.108 s
16/04/29 10:12:32 INFO DAGScheduler: Job 0 finished: collect at JavaLogQuery.java:112, took 0.850227 s
(10.10.10.10,"FRED",GET http://images.com/2013/Generic.jpg HTTP/1.1)    bytes=621   n=2
16/04/29 10:12:56 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:36394 in memory (size: 1746.0 B, free: 517.4 MB)

这里写图片描述
这里写图片描述
这里写图片描述
这里写图片描述

结束

16/04/29 10:12:56 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:36394 in memory (size: 1746.0 B, free: 517.4 MB)
16/04/29 10:16:13 INFO ContextCleaner: Cleaned accumulator 2
16/04/29 10:16:13 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:36394 in memory (size: 1897.0 B, free: 517.4 MB)
16/04/29 10:16:13 INFO ContextCleaner: Cleaned accumulator 1
16/04/29 10:24:29 WARN QueuedThreadPool: 5 threads could not be stopped
16/04/29 10:24:29 INFO SparkUI: Stopped Spark web UI at http://10.170.26.123:4040
16/04/29 10:24:29 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/04/29 10:24:29 INFO MemoryStore: MemoryStore cleared
16/04/29 10:24:29 INFO BlockManager: BlockManager stopped
16/04/29 10:24:29 INFO BlockManagerMaster: BlockManagerMaster stopped
16/04/29 10:24:30 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/04/29 10:24:30 INFO SparkContext: Successfully stopped SparkContext
16/04/29 10:24:30 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/04/29 10:24:30 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/04/29 10:24:30 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down.
  1. 总结
    java的代码实现spark API虽然代码冗余很多，但是很清楚显示了spark的执行过程，先比于scala的代码，较为清楚，而且java的代码和其他的项目结合效果可能好些。
文章标签：
日志服务
.NET
Java
分布式计算
监控
Apache
关键词：
apache spark实践
日志服务查询
apache spark查询
日志服务spark
apache spark日志
相关实践学习
日志服务之使用Nginx模式采集日志
本文介绍如何通过日志服务控制台创建Nginx模式的Logtail配置快速采集Nginx日志并进行多维度分析。
小金子
目录
相关文章
技术蜜糖罐
|
6天前
|
SQL 监控 关系型数据库
深入理解MySQL日志：通用查询、慢查询和错误日志详解
深入理解MySQL日志：通用查询、慢查询和错误日志详解
技术蜜糖罐
373 0
诸葛子房
|
6天前
|
分布式计算 监控 Spark
Spark 任务运行时日志分析
Spark 任务运行时日志分析
诸葛子房
55 0
阿里云云原生
|
6天前
|
存储 监控 数据可视化
无需重新学习，使用 Kibana 查询/可视化 SLS 数据
本文演示了使用 Kibana 连接 SLS ES 兼容接口进行查询和分析的方法。
阿里云云原生
65246 9
FrancekChen
|
1天前
|
分布式计算 Shell 开发工具
Spark编程实验二：RDD编程初级实践
Spark编程实验二：RDD编程初级实践
FrancekChen
7 1
阿里云瑶池数据库SelectDB
|
6天前
|
存储 监控 Apache
查询提速11倍、资源节省70%，阿里云数据库内核版 Apache Doris 在网易日志和时序场景的实践
网易的灵犀办公和云信利用 Apache Doris 改进了大规模日志和时序数据处理，取代了 Elasticsearch 和 InfluxDB。Doris 实现了更低的服务器资源消耗和更高的查询性能，相比 Elasticsearch，查询速度提升至少 11 倍，存储资源节省达 70%。Doris 的列式存储、高压缩比和倒排索引等功能，优化了日志和时序数据的存储与分析，降低了存储成本并提高了查询效率。在灵犀办公和云信的实际应用中，Doris 显示出显著的性能优势，成功应对了数据增长带来的挑战。
阿里云瑶池数据库SelectDB
1133 1
查询提速11倍、资源节省70%，阿里云数据库内核版 Apache Doris 在网易日志和时序场景的实践
xijie.xu
|
6天前
|
SQL 调度 Swift
【深入浅出】阿里自研开源搜索引擎Havenask日志查询
本次分享内容为Havenask的日志查询，文章包含了具体查询步骤和举例、实操演示，希望可以帮助大家更好的使用Havenask。
xijie.xu
1083 0
技术小达人
|
6天前
|
存储 监控 安全
带你读《Apache Doris 案例集》——07查询平均提速700% ，奇安信基于 Apache Doris 升级日志安全分析系统（1）
带你读《Apache Doris 案例集》——07查询平均提速700% ，奇安信基于 Apache Doris 升级日志安全分析系统（1）
技术小达人
183 1
技术小达人
|
6天前
|
SQL 存储 安全
带你读《Apache Doris 案例集》——07查询平均提速700% ，奇安信基于 Apache Doris 升级日志安全分析系统（2）
带你读《Apache Doris 案例集》——07查询平均提速700% ，奇安信基于 Apache Doris 升级日志安全分析系统（2）
技术小达人
213 0
阿里云云原生
|
6天前
|
SQL 存储 监控
SLS 查询新范式：使用 SPL 对日志进行交互式探索
像 Unix 命令一样支持多级管道级联，像加工预览一样实时处理查询结果，更便捷的交互，更丰富的算子，更灵活的探索半结构化日志，快来试试使用 SPL 语言查询日志数据吧~
阿里云云原生
46582 6
洛神灬殇
|
6天前
|
存储 移动开发 安全
【Linux技术专题】「必备基础知识」带你仔细梳理一下平时排查问题查询日志的基本操作和指令
【Linux技术专题】「必备基础知识」带你仔细梳理一下平时排查问题查询日志的基本操作和指令
洛神灬殇
25 0

热门文章

最新文章

  • 1
    工具变量法(两阶段最小二乘法2SLS)线性模型分析人均食品消费时间序列数据和回归诊断（下）
  • 2
    R语言计量经济学：工具变量法(两阶段最小二乘法2SLS)线性模型分析人均食品消费时间序列数据和回归诊断
  • 3
    基于Java的公司员工工作日志办公系统的设计与实现(源码+lw+部署文档+讲解等)
  • 4
    log4j异常日志过滤规则配置
  • 5
    通过AOP记录操作日志：提升应用可追踪性与安全性的利器
  • 6
    [SpringAop + Logback +MDC] 现网必备全链路日志追踪
  • 7
    Oracle日志文件：数据王国的“记事本”
  • 8
    【SpringBoot】讲清楚日志文件&&lombok
  • 9
    工具变量法(两阶段最小二乘法2SLS)线性模型分析人均食品消费时间序列数据和回归诊断2
  • 10
    双击eclipse提示an error has occurred See the log file E:\javatool\eclipse-mars-2\configuration\156991769
  • 1
    Spark学习---SparkSQL（概述、编程、数据的加载和保存、自定义UDFA、项目实战）
    145
  • 2
    安装Spark
    39
  • 3
    Spark学习---day07、Spark内核（Shuffle、任务执行）
    44
  • 4
    Spark读取变更Hudi数据集Schema实现分析
    53
  • 5
    实战|使用Spark Streaming写入Hudi
    72
  • 6
    adb spark的lakehouse api访问内表数据，还支持算子下推吗
    108
  • 7
    Spark学习---day06、Spark内核（源码提交流程、任务执行）
    45
  • 8
    Spark学习--day05、SparkCore电商网站实操、SparkCore-工程代码
    72
  • 9
    Spark【Spark学习大纲】简介+生态+RDD+安装+使用（xmind分享）
    36
  • 10
    Spark【基础知识 03】【RDD常用算子详解】（图片来源于网络）
    60

    • 相关课程

    更多
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第一阶段
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第二阶段
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第四阶段
  • 大数据实战项目：反爬虫系统（Lua+Spark+Redis+Hadoop框架搭建）第五阶段
  • 场景实践-基于阿里云Quick BI 对MOOC网站日志分析
  • 大数据实时计算框架Spark快速入门

    • 相关电子书

    更多
  • Hybrid Cloud and Apache Spark
  • Scalable Deep Learning on Spark
  • Comparison of Spark SQL with Hive

    • 相关实验场景

    更多
  • 使用CloudLens管理SLS日志采集
  • 使用CloudLens采集PolarDB日志并进行审计分析
  • 使用CloudLens采集RDS日志并进行审计分析
  • 基于SLS快速实现业务可观测
  • 使用日志服务SLS进行OSS可观测分析
  • 多账号体系下使用日志服务中心化的管理日志
    • 下一篇
    2024年阿里云免费云服务器及学生云服务器申请教程参考