Maven

#Maven#

已有1人关注此标签

内容分类

宋淑婷

AWS EMR Spark提交选项 - 失败

我试图在EMRSpark集群上运行python作业。 在spark submit-options中,我只使用以下设置来更改内存: master yarn --driver-memory 4g --executor-memory 2g 但是,每当我尝试提交作业时,它总是失败。 如何纠正Spark提交选项? 并且错误日志如下: Error: Unrecognized option: -- Usage: spark-submit [options] [app arguments]Usage: spark-submit --kill [submission ID] --master [spark://...]Usage: spark-submit --status [submission ID] --master [spark://...]Usage: spark-submit run-example [options] example-class [example args] Options: --master MASTER_URL spark://host:port, mesos://host:port, yarn, k8s://https://host:port, or local (Default: local[*]). --deploy-mode DEPLOY_MODE Whether to launch the driver program locally ("client") or on one of the worker machines inside the cluster ("cluster") (Default: client). --class CLASS_NAME Your application's main class (for Java / Scala apps). --name NAME A name of your application. --jars JARS Comma-separated list of jars to include on the driver and executor classpaths. --packages Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version. --exclude-packages Comma-separated list of groupId:artifactId, to exclude while resolving the dependencies provided in --packages to avoid dependency conflicts. --repositories Comma-separated list of additional remote repositories to search for the maven coordinates given with --packages. --py-files PY_FILES Comma-separated list of .zip, .egg, or .py files to place on the PYTHONPATH for Python apps. --files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName). --conf PROP=VALUE Arbitrary Spark configuration property. --properties-file FILE Path to a file from which to load extra properties. If not specified, this will look for conf/spark-defaults.conf. --driver-memory MEM Memory for driver (e.g. 1000M, 2G) (Default: 1024M). --driver-java-options Extra Java optionCommand exiting with ret '1'

宋淑婷

在EMR类中找不到Spark-submit异常java

我有一个jar包捆绑创建一个弹簧启动应用程序,其中我创建spark会话来做一些操作。 然后我发送这个jar AWS EMR,要么运行的spark-submit或者java -jar未工作的可执行文件。 这是我的jar结构 jar -tf META-INF/META-INF/MANIFEST.MForg/org/springframework/org/springframework/boot/org/springframework/boot/loader/org/springframework/boot/loader/data/org/springframework/boot/loader/data/RandomAccessDataFile$DataInputStream.classorg/springframework/boot/loader/jar/org/springframework/boot/loader/jar/CentralDirectoryEndRecord.classorg/springframework/boot/loader/data/RandomAccessDataFile$FileAccess.classorg/springframework/boot/loader/jar/CentralDirectoryParser.classorg/springframework/boot/loader/data/RandomAccessDataFile$1.classorg/springframework/boot/loader/jar/ZipInflaterInputStream.classorg/springframework/boot/loader/data/RandomAccessDataFile.classorg/springframework/boot/loader/jar/Handler.classorg/springframework/boot/loader/data/RandomAccessData.classorg/springframework/boot/loader/archive/org/springframework/boot/loader/archive/Archive$EntryFilter.classorg/springframework/boot/loader/jar/Bytes.classorg/springframework/boot/loader/archive/Archive.classorg/springframework/boot/loader/archive/JarFileArchive.classorg/springframework/boot/loader/PropertiesLauncher$PrefixMatchingArchiveFilter.classorg/springframework/boot/loader/PropertiesLauncher$ArchiveEntryFilter.classorg/springframework/boot/loader/PropertiesLauncher$1.classorg/springframework/boot/loader/PropertiesLauncher.classorg/springframework/boot/loader/Launcher.classorg/springframework/boot/loader/MainMethodRunner.classorg/springframework/boot/loader/ExecutableArchiveLauncher.classorg/springframework/boot/loader/WarLauncher.classorg/springframework/boot/loader/JarLauncher.classorg/springframework/boot/loader/util/org/springframework/boot/loader/util/SystemPropertyUtils.classorg/springframework/boot/loader/LaunchedURLClassLoader$UseFastConnectionExceptionsEnumeration.classorg/springframework/boot/loader/LaunchedURLClassLoader.classorg/springframework/boot/loader/jar/JarURLConnection$1.classorg/springframework/boot/loader/archive/JarFileArchive$EntryIterator.classorg/springframework/boot/loader/jar/JarURLConnection$JarEntryName.classorg/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator$EntryComparator.classorg/springframework/boot/loader/jar/JarURLConnection.classorg/springframework/boot/loader/jar/JarFile$1.classorg/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator.classorg/springframework/boot/loader/archive/ExplodedArchive$FileEntry.classorg/springframework/boot/loader/archive/ExplodedArchive$1.classorg/springframework/boot/loader/jar/JarFile$2.classorg/springframework/boot/loader/jar/JarFile$JarFileType.classorg/springframework/boot/loader/archive/ExplodedArchive.classorg/springframework/boot/loader/archive/JarFileArchive$JarFileEntry.classorg/springframework/boot/loader/jar/JarFile.classorg/springframework/boot/loader/jar/JarEntry.classorg/springframework/boot/loader/jar/FileHeader.classorg/springframework/boot/loader/jar/StringSequence.classorg/springframework/boot/loader/jar/AsciiBytes.classorg/springframework/boot/loader/jar/JarFileEntries$1.classorg/springframework/boot/loader/jar/JarFileEntries$EntryIterator.classorg/springframework/boot/loader/jar/JarFileEntries.classorg/springframework/boot/loader/jar/CentralDirectoryVisitor.classorg/springframework/boot/loader/jar/JarEntryFilter.classorg/springframework/boot/loader/jar/CentralDirectoryFileHeader.classorg/springframework/boot/loader/archive/Archive$Entry.classBOOT-INF/BOOT-INF/classes/BOOT-INF/classes/com/BOOT-INF/classes/com/ati/BOOT-INF/classes/com/ati/media/BOOT-INF/classes/com/ati/media/ref/BOOT-INF/classes/com/ati/media/ref/scd/BOOT-INF/classes/com/ati/media/ref/scd/config/BOOT-INF/classes/com/ati/media/ref/scd/controllers/BOOT-INF/classes/com/ati/media/ref/scd/error/BOOT-INF/classes/com/ati/media/ref/scd/repositories/BOOT-INF/classes/com/ati/media/ref/scd/services/BOOT-INF/classes/com/ati/media/ref/scd/services/validator/META-INF/maven/META-INF/maven/com.ati.media.ref/META-INF/maven/com.ati.media.ref/ref-svc-scd/BOOT-INF/classes/application-local.ymlBOOT-INF/classes/bootstrap.ymlBOOT-INF/classes/com/ati/media/ref/scd/config/SwaggerConfig.classBOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleExceptionHandler.classBOOT-INF/classes/com/ati/media/ref/scd/ScheduleApplication.classBOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleServiceImpl.classMETA-INF/maven/com.ati.media.ref/ref-svc-scd/pom.xmlBOOT-INF/classes/bootstrap-docker.ymlBOOT-INF/classes/com/ati/media/ref/scd/config/ScheduleConstants.classBOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleCreationException.classBOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.xmlBOOT-INF/classes/com/ati/media/ref/scd/services/ConfigurationService.classBOOT-INF/classes/git.propertiesMETA-INF/maven/com.ati.media.ref/ref-svc-scd/pom.propertiesBOOT-INF/classes/application.ymlBOOT-INF/classes/com/ati/media/ref/scd/config/SparkConfig.classBOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleEditException.classBOOT-INF/classes/com/ati/media/ref/scd/repositories/SparkAthenaRepository.classBOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleService.classMETA-INF/build-info.propertiesBOOT-INF/classes/bootstrap-local.ymlBOOT-INF/classes/com/ati/media/ref/scd/controllers/ScheduleController.classBOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.classBOOT-INF/classes/com/ati/media/ref/scd/services/AthenaQueryBuilder.classBOOT-INF/classes/com/ati/media/ref/scd/services/validator/ScheduleValidator.classMETA-INF/build.propertiesBOOT-INF/lib/BOOT-INF/lib/core-tech-starter-web-base-1.0.0.jarBOOT-INF/lib/spring-boot-starter-web-2.0.3.RELEASE.jarBOOT-INF/lib/spring-boot-starter-json-2.0.3.RELEASE.jarBOOT-INF/lib/jackson-datatype-jdk8-2.9.6.jarBOOT-INF/lib/jackson-datatype-jsr310-2.9.6.jarBOOT-INF/lib/jackson-module-parameter-names-2.9.6.jarBOOT-INF/lib/spring-boot-starter-tomcat-2.0.3.RELEASE.jarBOOT-INF/lib/tomcat-embed-core-8.5.31.jarBOOT-INF/lib/tomcat-embed-el-8.5.31.jarBOOT-INF/lib/tomcat-embed-websocket-8.5.31.jarBOOT-INF/lib/hibernate-validator-6.0.10.Final.jarBOOT-INF/lib/spring-web-5.0.7.RELEASE.jarBOOT-INF/lib/spring-webmvc-5.0.7.RELEASE.jarBOOT-INF/lib/spring-expression-5.0.7.RELEASE.jarBOOT-INF/lib/spring-boot-starter-webflux-2.0.3.RELEASE.jarBOOT-INF/lib/spring-boot-starter-reactor-netty-2.0.3.RELEASE.jarBOOT-INF/lib/reactor-netty-0.7.8.RELEASE.jarBOOT-INF/lib/netty-codec-http-4.1.25.Final.jarBOOT-INF/lib/netty-codec-4.1.25.Final.jarBOOT-INF/lib/netty-handler-4.1.25.Final.jarBOOT-INF/lib/netty-buffer-4.1.25.Final.jarBOOT-INF/lib/netty-transport-4.1.25.Final.jarBOOT-INF/lib/netty-resolver-4.1.25.Final.jarBOOT-INF/lib/netty-handler-proxy-4.1.25.Final.jarBOOT-INF/lib/netty-codec-socks-4.1.25.Final.jarBOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jarBOOT-INF/lib/netty-common-4.1.25.Final.jarBOOT-INF/lib/netty-transport-native-unix-common-4.1.25.Final.jarBOOT-INF/lib/spring-webflux-5.0.7.RELEASE.jarBOOT-INF/lib/reactor-core-3.1.8.RELEASE.jarBOOT-INF/lib/nio-multipart-parser-1.1.0.jarBOOT-INF/lib/nio-stream-storage-1.1.3.jarBOOT-INF/lib/spring-boot-starter-actuator-2.0.3.RELEASE.jarBOOT-INF/lib/spring-boot-actuator-autoconfigure-2.0.3.RELEASE.jarBOOT-INF/lib/spring-boot-actuator-2.0.3.RELEASE.jarBOOT-INF/lib/micrometer-core-1.0.5.jarBOOT-INF/lib/HdrHistogram-2.1.10.jarBOOT-INF/lib/LatencyUtils-2.0.3.jarBOOT-INF/lib/spring-boot-starter-log4j2-2.0.3.RELEASE.jarBOOT-INF/lib/log4j-slf4j-impl-2.10.0.jarBOOT-INF/lib/log4j-core-2.10.0.jarBOOT-INF/lib/log4j-jul-2.10.0.jarBOOT-INF/lib/log4j-1.2-api-2.7.jarBOOT-INF/lib/log4j-api-2.10.0.jarBOOT-INF/lib/lombok-1.18.0.jarBOOT-INF/lib/micrometer-registry-prometheus-1.0.3.jarBOOT-INF/lib/simpleclient_common-0.3.0.jarBOOT-INF/lib/simpleclient-0.3.0.jarBOOT-INF/lib/spring-cloud-starter-zipkin-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-starter-sleuth-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-sleuth-core-2.0.0.RELEASE.jarBOOT-INF/lib/aspectjrt-1.8.13.jarBOOT-INF/lib/brave-5.1.0.jarBOOT-INF/lib/brave-context-log4j2-5.1.0.jarBOOT-INF/lib/brave-instrumentation-spring-web-5.1.0.jarBOOT-INF/lib/brave-instrumentation-http-5.1.0.jarBOOT-INF/lib/brave-instrumentation-spring-rabbit-5.1.0.jarBOOT-INF/lib/brave-instrumentation-kafka-clients-5.1.0.jarBOOT-INF/lib/brave-instrumentation-httpclient-5.1.0.jarBOOT-INF/lib/brave-instrumentation-httpasyncclient-5.1.0.jarBOOT-INF/lib/brave-instrumentation-spring-webmvc-5.1.0.jarBOOT-INF/lib/brave-instrumentation-servlet-5.1.0.jarBOOT-INF/lib/spring-cloud-sleuth-zipkin-2.0.0.RELEASE.jarBOOT-INF/lib/zipkin-2.9.3.jarBOOT-INF/lib/zipkin-reporter-2.7.3.jarBOOT-INF/lib/zipkin-sender-kafka11-2.7.3.jarBOOT-INF/lib/zipkin-sender-amqp-client-2.7.3.jarBOOT-INF/lib/core-tech-starter-amqp-1.0.0.jarBOOT-INF/lib/spring-cloud-stream-2.0.0.RELEASE.jarBOOT-INF/lib/spring-boot-starter-validation-2.0.3.RELEASE.jarBOOT-INF/lib/spring-messaging-5.0.7.RELEASE.jarBOOT-INF/lib/spring-integration-core-5.0.6.RELEASE.jarBOOT-INF/lib/spring-integration-jmx-5.0.6.RELEASE.jarBOOT-INF/lib/spring-tuple-1.0.0.RELEASE.jarBOOT-INF/lib/spring-integration-tuple-1.0.0.RELEASE.jarBOOT-INF/lib/spring-boot-starter-amqp-2.0.3.RELEASE.jarBOOT-INF/lib/spring-rabbit-2.0.4.RELEASE.jarBOOT-INF/lib/spring-amqp-2.0.4.RELEASE.jarBOOT-INF/lib/amqp-client-5.1.2.jarBOOT-INF/lib/spring-cloud-stream-binder-rabbit-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-stream-binder-rabbit-core-2.0.0.RELEASE.jarBOOT-INF/lib/spring-integration-amqp-5.0.6.RELEASE.jarBOOT-INF/lib/spring-cloud-starter-config-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-starter-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-context-2.0.0.RELEASE.jarBOOT-INF/lib/spring-security-rsa-1.0.5.RELEASE.jarBOOT-INF/lib/bcpkix-jdk15on-1.56.jarBOOT-INF/lib/spring-cloud-config-client-2.0.0.RELEASE.jarBOOT-INF/lib/spring-boot-starter-aop-2.0.3.RELEASE.jarBOOT-INF/lib/spring-aop-5.0.7.RELEASE.jarBOOT-INF/lib/aspectjweaver-1.8.13.jarBOOT-INF/lib/core-tech-lib-core-1.0.0.jarBOOT-INF/lib/spring-retry-1.2.0.RELEASE.jarBOOT-INF/lib/spring-cloud-starter-openfeign-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-openfeign-core-2.0.0.RELEASE.jarBOOT-INF/lib/feign-core-9.5.1.jarBOOT-INF/lib/feign-slf4j-9.5.1.jarBOOT-INF/lib/feign-hystrix-9.5.1.jarBOOT-INF/lib/feign-java8-9.5.1.jarBOOT-INF/lib/spring-cloud-starter-netflix-ribbon-2.0.0.RELEASE.jarBOOT-INF/lib/ribbon-2.2.5.jarBOOT-INF/lib/ribbon-transport-2.2.5.jarBOOT-INF/lib/rxnetty-contexts-0.4.9.jarBOOT-INF/lib/rxnetty-servo-0.4.9.jarBOOT-INF/lib/rxnetty-0.4.9.jarBOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final.jarBOOT-INF/lib/ribbon-core-2.2.5.jarBOOT-INF/lib/ribbon-httpclient-2.2.5.jarBOOT-INF/lib/jersey-client-1.19.1.jarBOOT-INF/lib/jersey-apache-client4-1.19.1.jarBOOT-INF/lib/servo-core-0.12.21.jarBOOT-INF/lib/netflix-commons-util-0.3.0.jarBOOT-INF/lib/ribbon-loadbalancer-2.2.5.jarBOOT-INF/lib/netflix-statistics-0.1.1.jarBOOT-INF/lib/commons-text-1.3.jarBOOT-INF/lib/commons-collections4-4.1.jarBOOT-INF/lib/core-tech-starter-web-api-1.0.0.jarBOOT-INF/lib/springfox-swagger2-2.9.2.jarBOOT-INF/lib/swagger-annotations-1.5.20.jarBOOT-INF/lib/swagger-models-1.5.20.jarBOOT-INF/lib/springfox-spi-2.9.2.jarBOOT-INF/lib/springfox-core-2.9.2.jarBOOT-INF/lib/springfox-schema-2.9.2.jarBOOT-INF/lib/springfox-swagger-common-2.9.2.jarBOOT-INF/lib/springfox-spring-web-2.9.2.jarBOOT-INF/lib/guava-20.0.jarBOOT-INF/lib/classmate-1.3.4.jarBOOT-INF/lib/spring-plugin-core-1.2.0.RELEASE.jarBOOT-INF/lib/spring-plugin-metadata-1.2.0.RELEASE.jarBOOT-INF/lib/mapstruct-1.2.0.Final.jarBOOT-INF/lib/springfox-swagger-ui-2.9.2.jarBOOT-INF/lib/spring-cloud-starter-netflix-hystrix-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-netflix-core-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-netflix-ribbon-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-netflix-archaius-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-starter-netflix-archaius-2.0.0.RELEASE.jarBOOT-INF/lib/archaius-core-0.7.6.jarBOOT-INF/lib/hystrix-core-1.5.12.jarBOOT-INF/lib/rxjava-1.3.8.jarBOOT-INF/lib/hystrix-serialization-1.5.12.jarBOOT-INF/lib/jackson-module-afterburner-2.9.6.jarBOOT-INF/lib/hystrix-metrics-event-stream-1.5.12.jarBOOT-INF/lib/hystrix-javanica-1.5.12.jarBOOT-INF/lib/asm-5.0.4.jarBOOT-INF/lib/rxjava-reactive-streams-1.2.1.jarBOOT-INF/lib/reactive-streams-1.0.2.jarBOOT-INF/lib/spring-cloud-netflix-hystrix-stream-2.0.0.RELEASE.jarBOOT-INF/lib/spring-cloud-commons-2.0.0.RELEASE.jarBOOT-INF/lib/spring-security-crypto-5.0.6.RELEASE.jarBOOT-INF/lib/spring-cloud-starter-stream-rabbit-2.0.0.RELEASE.jarBOOT-INF/lib/postgresql-42.2.2.jarBOOT-INF/lib/athenaJDBC-1.1.0.jarBOOT-INF/lib/lib-client-scd-0.0.0.jarBOOT-INF/lib/core-tech-starter-web-client-1.0.0.jarBOOT-INF/lib/hibernate-jpa-2.1-api-1.0.2.Final.jarBOOT-INF/lib/gson-2.8.5.jarBOOT-INF/lib/spring-boot-starter-data-jpa-2.0.3.RELEASE.jarBOOT-INF/lib/spring-boot-starter-2.0.3.RELEASE.jarBOOT-INF/lib/spring-boot-2.0.3.RELEASE.jarBOOT-INF/lib/spring-boot-autoconfigure-2.0.3.RELEASE.jarBOOT-INF/lib/javax.annotation-api-1.3.2.jarBOOT-INF/lib/snakeyaml-1.19.jarBOOT-INF/lib/spring-boot-starter-jdbc-2.0.3.RELEASE.jarBOOT-INF/lib/HikariCP-2.7.9.jarBOOT-INF/lib/spring-jdbc-5.0.7.RELEASE.jarBOOT-INF/lib/hibernate-core-5.2.17.Final.jarBOOT-INF/lib/jboss-logging-3.3.2.Final.jarBOOT-INF/lib/javassist-3.22.0-GA.jarBOOT-INF/lib/antlr-2.7.7.jarBOOT-INF/lib/jandex-2.0.3.Final.jarBOOT-INF/lib/dom4j-1.6.1.jarBOOT-INF/lib/hibernate-commons-annotations-5.0.1.Final.jarBOOT-INF/lib/javax.transaction-api-1.2.jarBOOT-INF/lib/spring-data-jpa-2.0.8.RELEASE.jarBOOT-INF/lib/spring-data-commons-2.0.8.RELEASE.jarBOOT-INF/lib/spring-orm-5.0.7.RELEASE.jarBOOT-INF/lib/spring-context-5.0.7.RELEASE.jarBOOT-INF/lib/spring-beans-5.0.7.RELEASE.jarBOOT-INF/lib/spring-aspects-5.0.7.RELEASE.jarBOOT-INF/lib/mybatis-spring-boot-starter-1.3.1.jarBOOT-INF/lib/mybatis-spring-boot-autoconfigure-1.3.1.jarBOOT-INF/lib/mybatis-3.4.5.jarBOOT-INF/lib/mybatis-spring-1.3.1.jarBOOT-INF/lib/spring-boot-starter-jooq-2.0.3.RELEASE.jarBOOT-INF/lib/spring-tx-5.0.7.RELEASE.jarBOOT-INF/lib/jooq-3.10.7.jarBOOT-INF/lib/byte-buddy-1.7.11.jarBOOT-INF/lib/objenesis-2.6.jarBOOT-INF/lib/spring-core-5.0.7.RELEASE.jarBOOT-INF/lib/spring-jcl-5.0.7.RELEASE.jarBOOT-INF/lib/core-tech-starter-web-request-details-1.0.0.jarBOOT-INF/lib/spark-core_2.11-2.3.0.jarBOOT-INF/lib/avro-1.7.7.jarBOOT-INF/lib/jackson-core-asl-1.9.13.jarBOOT-INF/lib/paranamer-2.3.jarBOOT-INF/lib/commons-compress-1.4.1.jarBOOT-INF/lib/xz-1.0.jarBOOT-INF/lib/avro-mapred-1.7.7-hadoop2.jarBOOT-INF/lib/avro-ipc-1.7.7.jarBOOT-INF/lib/avro-ipc-1.7.7-tests.jarBOOT-INF/lib/chill_2.11-0.8.4.jarBOOT-INF/lib/kryo-shaded-3.0.3.jarBOOT-INF/lib/minlog-1.3.0.jarBOOT-INF/lib/chill-java-0.8.4.jarBOOT-INF/lib/xbean-asm5-shaded-4.4.jarBOOT-INF/lib/hadoop-client-2.6.5.jarBOOT-INF/lib/hadoop-hdfs-2.6.5.jarBOOT-INF/lib/xercesImpl-2.9.1.jarBOOT-INF/lib/xml-apis-1.4.01.jarBOOT-INF/lib/htrace-core-3.0.4.jarBOOT-INF/lib/hadoop-mapreduce-client-app-2.6.5.jarBOOT-INF/lib/hadoop-mapreduce-client-common-2.6.5.jarBOOT-INF/lib/hadoop-mapreduce-client-shuffle-2.6.5.jarBOOT-INF/lib/hadoop-mapreduce-client-core-2.6.5.jarBOOT-INF/lib/hadoop-mapreduce-client-jobclient-2.6.5.jarBOOT-INF/lib/hadoop-annotations-2.6.5.jarBOOT-INF/lib/spark-launcher_2.11-2.3.0.jarBOOT-INF/lib/spark-kvstore_2.11-2.3.0.jarBOOT-INF/lib/leveldbjni-all-1.8.jarBOOT-INF/lib/jackson-core-2.9.6.jarBOOT-INF/lib/spark-network-common_2.11-2.3.0.jarBOOT-INF/lib/spark-network-shuffle_2.11-2.3.0.jarBOOT-INF/lib/spark-unsafe_2.11-2.3.0.jarBOOT-INF/lib/jets3t-0.9.4.jarBOOT-INF/lib/httpcore-4.4.9.jarBOOT-INF/lib/activation-1.1.1.jarBOOT-INF/lib/bcprov-jdk15on-1.52.jarBOOT-INF/lib/java-xmlbuilder-1.1.jarBOOT-INF/lib/base64-2.3.8.jarBOOT-INF/lib/curator-recipes-4.0.1.jarBOOT-INF/lib/curator-framework-4.0.1.jarBOOT-INF/lib/javax.servlet-api-3.1.0.jarBOOT-INF/lib/commons-lang3-3.7.jarBOOT-INF/lib/commons-math3-3.4.1.jarBOOT-INF/lib/jsr305-1.3.9.jarBOOT-INF/lib/slf4j-api-1.7.25.jarBOOT-INF/lib/jul-to-slf4j-1.7.25.jarBOOT-INF/lib/jcl-over-slf4j-1.7.25.jarBOOT-INF/lib/compress-lzf-1.0.3.jarBOOT-INF/lib/snappy-java-1.1.2.6.jarBOOT-INF/lib/lz4-java-1.4.0.jarBOOT-INF/lib/zstd-jni-1.3.2-2.jarBOOT-INF/lib/RoaringBitmap-0.5.11.jarBOOT-INF/lib/commons-net-2.2.jarBOOT-INF/lib/scala-library-2.11.8.jarBOOT-INF/lib/json4s-jackson_2.11-3.2.11.jarBOOT-INF/lib/json4s-core_2.11-3.2.11.jarBOOT-INF/lib/json4s-ast_2.11-3.2.11.jarBOOT-INF/lib/scalap-2.11.0.jarBOOT-INF/lib/scala-compiler-2.11.0.jarBOOT-INF/lib/scala-xml_2.11-1.0.1.jarBOOT-INF/lib/jersey-client-2.26.jarBOOT-INF/lib/javax.ws.rs-api-2.1.jarBOOT-INF/lib/javax.inject-2.5.0-b42.jarBOOT-INF/lib/jersey-common-2.26.jarBOOT-INF/lib/osgi-resource-locator-1.0.1.jarBOOT-INF/lib/jersey-server-2.26.jarBOOT-INF/lib/jersey-media-jaxb-2.26.jarBOOT-INF/lib/validation-api-2.0.1.Final.jarBOOT-INF/lib/jersey-container-servlet-2.26.jarBOOT-INF/lib/jersey-container-servlet-core-2.26.jarBOOT-INF/lib/netty-all-4.1.25.Final.jarBOOT-INF/lib/netty-3.9.9.Final.jarBOOT-INF/lib/stream-2.7.0.jarBOOT-INF/lib/metrics-core-3.2.6.jarBOOT-INF/lib/metrics-jvm-3.2.6.jarBOOT-INF/lib/metrics-json-3.2.6.jarBOOT-INF/lib/metrics-graphite-3.2.6.jarBOOT-INF/lib/jackson-databind-2.9.6.jarBOOT-INF/lib/jackson-module-scala_2.11-2.9.6.jarBOOT-INF/lib/scala-reflect-2.11.11.jarBOOT-INF/lib/jackson-module-paranamer-2.9.6.jarBOOT-INF/lib/ivy-2.4.0.jarBOOT-INF/lib/oro-2.0.8.jarBOOT-INF/lib/pyrolite-4.13.jarBOOT-INF/lib/py4j-0.10.6.jarBOOT-INF/lib/spark-tags_2.11-2.3.0.jarBOOT-INF/lib/commons-crypto-1.0.0.jarBOOT-INF/lib/unused-1.0.0.jarBOOT-INF/lib/simba-athena-jdbc-driver-2.0.2.jarBOOT-INF/lib/spark-sql_2.11-2.3.0.jarBOOT-INF/lib/univocity-parsers-2.5.9.jarBOOT-INF/lib/spark-sketch_2.11-2.3.0.jarBOOT-INF/lib/spark-catalyst_2.11-2.3.0.jarBOOT-INF/lib/scala-parser-combinators_2.11-1.0.4.jarBOOT-INF/lib/janino-3.0.8.jarBOOT-INF/lib/commons-compiler-3.0.8.jarBOOT-INF/lib/antlr4-runtime-4.7.jarBOOT-INF/lib/orc-core-1.4.1-nohive.jarBOOT-INF/lib/protobuf-java-2.5.0.jarBOOT-INF/lib/commons-lang-2.6.jarBOOT-INF/lib/aircompressor-0.8.jarBOOT-INF/lib/orc-mapreduce-1.4.1-nohive.jarBOOT-INF/lib/parquet-column-1.8.2.jarBOOT-INF/lib/parquet-common-1.8.2.jarBOOT-INF/lib/parquet-encoding-1.8.2.jarBOOT-INF/lib/parquet-hadoop-1.8.2.jarBOOT-INF/lib/parquet-format-2.3.1.jarBOOT-INF/lib/parquet-jackson-1.8.2.jarBOOT-INF/lib/arrow-vector-0.8.0.jarBOOT-INF/lib/arrow-format-0.8.0.jarBOOT-INF/lib/arrow-memory-0.8.0.jarBOOT-INF/lib/hppc-0.7.2.jarBOOT-INF/lib/flatbuffers-1.2.0-3f79e055.jarBOOT-INF/lib/spark-yarn_2.11-2.3.0.jarBOOT-INF/lib/hadoop-yarn-api-2.6.5.jarBOOT-INF/lib/hadoop-yarn-common-2.6.5.jarBOOT-INF/lib/jaxb-api-2.3.0.jarBOOT-INF/lib/jetty-util-6.1.26.jarBOOT-INF/lib/jackson-jaxrs-1.9.13.jarBOOT-INF/lib/jackson-xc-1.9.13.jarBOOT-INF/lib/commons-cli-1.2.jarBOOT-INF/lib/guice-servlet-3.0.jarBOOT-INF/lib/commons-io-2.4.jarBOOT-INF/lib/guice-3.0.jarBOOT-INF/lib/javax.inject-1.jarBOOT-INF/lib/aopalliance-1.0.jarBOOT-INF/lib/log4j-1.2.17.jarBOOT-INF/lib/hadoop-yarn-server-web-proxy-2.6.5.jarBOOT-INF/lib/hadoop-yarn-server-common-2.6.5.jarBOOT-INF/lib/jetty-6.1.26.jarBOOT-INF/lib/hadoop-yarn-client-2.6.5.jarBOOT-INF/lib/hadoop-aws-2.7.3.jarBOOT-INF/lib/hadoop-common-2.7.3.jarBOOT-INF/lib/xmlenc-0.52.jarBOOT-INF/lib/commons-collections-3.2.2.jarBOOT-INF/lib/servlet-api-2.5.jarBOOT-INF/lib/jsp-api-2.1.jarBOOT-INF/lib/jersey-core-1.19.1.jarBOOT-INF/lib/jsr311-api-1.1.1.jarBOOT-INF/lib/jersey-json-1.9.jarBOOT-INF/lib/jettison-1.1.jarBOOT-INF/lib/jaxb-impl-2.2.3-1.jarBOOT-INF/lib/jersey-server-1.19.1.jarBOOT-INF/lib/commons-logging-1.1.3.jarBOOT-INF/lib/commons-configuration-1.6.jarBOOT-INF/lib/commons-digester-1.8.jarBOOT-INF/lib/commons-beanutils-1.7.0.jarBOOT-INF/lib/commons-beanutils-core-1.8.0.jarBOOT-INF/lib/hadoop-auth-2.7.3.jarBOOT-INF/lib/apacheds-kerberos-codec-2.0.0-M15.jarBOOT-INF/lib/apacheds-i18n-2.0.0-M15.jarBOOT-INF/lib/api-asn1-api-1.0.0-M20.jarBOOT-INF/lib/api-util-1.0.0-M20.jarBOOT-INF/lib/jsch-0.1.54.jarBOOT-INF/lib/curator-client-2.7.1.jarBOOT-INF/lib/htrace-core-3.1.0-incubating.jarBOOT-INF/lib/zookeeper-3.4.6.jarBOOT-INF/lib/jackson-annotations-2.9.0.jarBOOT-INF/lib/aws-java-sdk-1.7.4.jarBOOT-INF/lib/spark-hive_2.11-2.3.0.jarBOOT-INF/lib/parquet-hadoop-bundle-1.6.0.jarBOOT-INF/lib/hive-exec-1.2.1.spark2.jarBOOT-INF/lib/javolution-5.5.1.jarBOOT-INF/lib/apache-log4j-extras-1.2.17.jarBOOT-INF/lib/antlr-runtime-3.4.jarBOOT-INF/lib/stringtemplate-3.2.1.jarBOOT-INF/lib/ST4-4.0.4.jarBOOT-INF/lib/JavaEWAH-0.3.2.jarBOOT-INF/lib/snappy-0.2.jarBOOT-INF/lib/stax-api-1.0.1.jarBOOT-INF/lib/opencsv-2.3.jarBOOT-INF/lib/hive-metastore-1.2.1.spark2.jarBOOT-INF/lib/bonecp-0.8.0.RELEASE.jarBOOT-INF/lib/datanucleus-api-jdo-3.2.6.jarBOOT-INF/lib/datanucleus-rdbms-3.2.9.jarBOOT-INF/lib/commons-pool-1.6.jarBOOT-INF/lib/commons-dbcp-1.4.jarBOOT-INF/lib/jdo-api-3.0.1.jarBOOT-INF/lib/jta-1.1.jarBOOT-INF/lib/commons-httpclient-3.1.jarBOOT-INF/lib/calcite-avatica-1.2.0-incubating.jarBOOT-INF/lib/calcite-core-1.2.0-incubating.jarBOOT-INF/lib/calcite-linq4j-1.2.0-incubating.jarBOOT-INF/lib/eigenbase-properties-1.1.5.jarBOOT-INF/lib/httpclient-4.5.5.jarBOOT-INF/lib/jackson-mapper-asl-1.9.13.jarBOOT-INF/lib/commons-codec-1.11.jarBOOT-INF/lib/joda-time-2.9.9.jarBOOT-INF/lib/jodd-core-3.5.2.jarBOOT-INF/lib/datanucleus-core-3.2.10.jarBOOT-INF/lib/libthrift-0.9.3.jarBOOT-INF/lib/libfb303-0.9.3.jarBOOT-INF/lib/derby-10.14.1.0.jarBOOT-INF/lib/core-tech-starter-docker-spring-boot-1.0.0.jar当试图运行 spark-submit --master yarn --deploy-mode client --driver-memory 10G --executor-memory 15G --class com.ati.media.ref.scd.ScheduleApplication有错误java.lang.ClassNotFoundException: com.ati.media.ref.scd.ScheduleApplication at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:235) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 当我尝试使用 java -jar 应用程序没有任何问题。 java -cp 导致与主要calss未找到的spark-submit相同的错误。 这里有什么问题。我有主要功能ScheduleApplication。

杨晓锋

group类型的仓库都无法展示详情,怎么能查看的到Maven依赖呢?

group类型的仓库都无法展示详情,怎么能查看的到Maven依赖呢?

李博bluemind

请教大神们一个问题,maven打包怎么把依赖包打进目标jar,但是不解压依赖包

请教大神们一个问题,maven打包怎么把依赖包打进目标jar,但是不解压依赖包

李博bluemind

maven怎么同时build thin jar和fat jar?

maven怎么同时build thin jar和fat jar?

社区小助手

编译spark2.4.0 遇到这个问题

编译spark2.4.0 遇到这个问题 [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-hive-thriftserver_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project spark-hive-thriftserver_2.10: Execution scala-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.

游客h5pdq2cukapjg

第一个flink的WordCount程序运行时错误,

错误: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/streaming/api/environment/StreamExecutionEnvironment at test.FlinkTest.main(FlinkTest.java:16) Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.api.environment.StreamExecutionEnvironment 就是这里错了: 'StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();' maven依赖有添加,不知道怎么回事

不喝酒的李白

springcloud maven项目 在本地用idea运行可以,放阿里云服务器报错

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.eurekaApplicationInfoManager' defined in class path resource [org/springframework/cloud/netflix/eureka/EurekaClientAutoConfiguration$RefreshableEurekaClientConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.appinfo.ApplicationInfoManager]: Factory method 'eurekaApplicationInfoManager' threw exception; nested exception is java.lang.NullPointerException    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:590) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1256) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1105) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:503) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390) ~[spring-cloud-context-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:184) ~[spring-cloud-context-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:350) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:35) ~[spring-aop-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672) ~[spring-aop-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at com.netflix.appinfo.ApplicationInfoManager $$ EnhancerBySpringCGLIB $$ 59479076.getInfo() ~[eureka-client-1.9.2.jar!/:1.9.2]    at org.springframework.cloud.netflix.eureka.serviceregistry.EurekaServiceRegistry.maybeInitializeClient(EurekaServiceRegistry.java:56) ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.cloud.netflix.eureka.serviceregistry.EurekaServiceRegistry.register(EurekaServiceRegistry.java:39) ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.cloud.netflix.eureka.serviceregistry.EurekaAutoServiceRegistration.start(EurekaAutoServiceRegistration.java:80) ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    ... 22 common frames omittedCaused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.appinfo.ApplicationInfoManager]: Factory method 'eurekaApplicationInfoManager' threw exception; nested exception is java.lang.NullPointerException    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:582) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    ... 38 common frames omittedCaused by: java.lang.NullPointerException: null    at java.lang.String.replace(String.java:2240) ~[na:1.8.0_201]    at com.netflix.appinfo.InstanceInfo$Builder.setStatusPageUrl(InstanceInfo.java:617) ~[eureka-client-1.9.2.jar!/:1.9.2]    at org.springframework.cloud.netflix.eureka.InstanceInfoFactory.create(InstanceInfoFactory.java:61) ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration$RefreshableEurekaClientConfiguration.eurekaApplicationInfoManager(EurekaClientAutoConfiguration.java:278) ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration$RefreshableEurekaClientConfiguration $$ EnhancerBySpringCGLIB $$ efe7787f.CGLIB$eurekaApplicationInfoManager$0() ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration$RefreshableEurekaClientConfiguration $$ EnhancerBySpringCGLIB $$ efe7787f $$ FastClassBySpringCGLIB $$ 49aa0745.invoke() ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361) ~[spring-context-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    at org.springframework.cloud.netflix.eureka.EurekaClientAutoConfiguration$RefreshableEurekaClientConfiguration $$ EnhancerBySpringCGLIB $$ efe7787f.eurekaApplicationInfoManager() ~[spring-cloud-netflix-eureka-client-2.0.0.RELEASE.jar!/:2.0.0.RELEASE]    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_201]    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_201]    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_201]    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_201]    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.0.7.RELEASE.jar!/:5.0.7.RELEASE]    ... 39 common frames omitted

赵慧

各位大牛下午好,请问在编译Blink时,发生如下异常: 我的maven版本:3.2.5,请问该如何解决 Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project flink-table_2.11: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error

各位大牛下午好,请问在编译Blink时,发生如下异常: 我的maven版本:3.2.5,请问该如何解决Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first) on project flink-table_2.11: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: -10000 (Exit value: -10000) -> [Help 1]我使用该命令进行编译:mvn clean package -Dmaven.test.skip=true -Dhadoop.version=2.7.3

李博bluemind

请问下gradle 和maven哪一个使用得比较多?

请问下gradle 和maven哪一个使用得比较多?

期待l

请教一下, 单纯的Protobuf只能直接实现序列化,而不能直接实现RPC吗?

maven 引入Protobuf core,只能直接用来序列化和反序列化吗, 而不能直接实现RPC ???? 比如像Avro那样直接使用对应的实现类,Avro实现TCP的就使用 SocketServer和SocketTransceiverProtobuf怎么实现???跪谢

苛矣

SonarQube bug检测结果控制Maven build是否成功

maven集成sonarqube,如果想通过sonarqube分析代码后的issue数量来控制maven的build是否成功,如果issue数超过我们设定的数量,则直接使maven build失败并终止。 想问下大家有做过类似的操作吗?或是大致提供下思路也行。小白一枚,非常感谢哈~

jack胡

李博bluemind

[@talishboy][¥20]Java 中,Maven 和 ANT 有什么区别?

问题来自Java技术沙龙的李博Java线下沙龙报名链接:https://yq.aliyun.com/activity/796

kill`losmart

【求助】scala运行时报错ArrayIndexOutOfBoundsException

主程序代码如下: import org.apache.spark.rdd.RDD import org.apache.spark.{SparkConf, SparkContext} //todo:利用scala语言开发spark的wordcount程序(本地运行) object WordCount { def main(args: Array[String]): Unit = { //1、创建SparkConf对象 设置applicationName和master地址 local[2]表示本地采用2个线程 val sparkConf: SparkConf = new SparkConf().setAppName("WordCount").setMaster("local[2]") //2、创建SparkContext对象,它是所有spark程序执行入口,它内部会构建DAGScheduler和TaskScheduler val sc = new SparkContext(sparkConf) //设置日志输出级别 sc.setLogLevel("warn") //3、读取文件数据 val data: RDD[String] = sc.textFile("E:\\words.txt") //4、切分每一行,获取所有的单词 val words: RDD[String] = data.flatMap(x=>x.split(" ")) //5、每个单词计为1 val wordAndOne: RDD[(String, Int)] = words.map(x =>(x,1)) //6、相同单词出现的1累加 val result: RDD[(String, Int)] = wordAndOne.reduceByKey((x:Int,y:Int)=>x+y) //按照单词出现的次数降序排列 默认第二个参数是true,在这里表示升序,改为false就是降序 val sortedRDD: RDD[(String, Int)] = result.sortBy(x => x._2,false) //7、收集数据打印 val finalResult: Array[(String, Int)] = sortedRDD.collect finalResult.foreach(x=>println(x)) //8、关闭sc sc.stop() } } pom.xml文件: <properties> <!--<scala.version>2.11.8</scala.version>--> <hadoop.version>3.1.0</hadoop.version> <!--<spark.version>2.1.3</spark.version>--> </properties> <dependencies> <dependency> <groupId>org.scala-lang</groupId> <artifactId>scala-library</artifactId> <version>2.12.7</version> </dependency> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>2.4.0</version> </dependency> </dependencies> <build> <sourceDirectory>src/main/scala</sourceDirectory> <testSourceDirectory>src/test/scala</testSourceDirectory> <plugins> <plugin> <groupId>net.alchim31.maven</groupId> <artifactId>scala-maven-plugin</artifactId> <version>3.2.2</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> <configuration> <args> <arg>-dependencyfile</arg> <arg>${project.build.directory}/.scala_dependencies</arg> </args> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.4.3</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <filters> <filter> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass></mainClass> </transformer> </transformers> </configuration> </execution> </executions> </plugin> </plugins> </build> </project> 程序启动后报错内容如下:

flink小助手

maven没有下载依赖项,因为trustAnchors参数必须是非空的

我在Virtual Box图像中安装了新的Linux Mint 19。因此,在安装Java 8之后,我下载了Eclipse Scala,并且我尝试添加到我的新项目的任何依赖项都不会被下载,并且在其pom.xml中会标记错误。 在我的Eclipse Scala for Windows(在主机上)检查网址并完全相同之后,它可以正常运行,但不适用于Linux。 我认为它应该是一个日食错误,因为它与证书有关。 如果它有帮助,我正在尝试编写一些Apache Flink程序,但无法用这个错误来做... 我附上错误: Failed to read artifact descriptor for org.apache.flink:flink-scala_2.11:jar:1.6.0 org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for org.apache.flink:flink-scala_2.11:jar:1.6.0 at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:282) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:198) at org.eclipse.aether.internal.impl.DefaultDependencyCollector.resolveCachedArtifactDescriptor(DefaultDependencyCollector.java:535) at org.eclipse.aether.internal.impl.DefaultDependencyCollector.getArtifactDescriptorResult(DefaultDependencyCollector.java:519) at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:409) at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:363) at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:351) at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:254) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:316) at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:172) at org.apache.maven.project.DefaultProjectBuilder.resolveDependencies(DefaultProjectBuilder.java:215) at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:188) at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:119) at org.eclipse.m2e.core.internal.embedder.MavenImpl.readMavenProject(MavenImpl.java:636) at org.eclipse.m2e.core.internal.project.registry.DefaultMavenDependencyResolver.resolveProjectDependencies(DefaultMavenDependencyResolver.java:63) at org.eclipse.m2e.core.internal.project.registry.ProjectRegistryManager.refreshPhase2(ProjectRegistryManager.java:530) at org.eclipse.m2e.core.internal.project.registry.ProjectRegistryManager$3.call(ProjectRegistryManager.java:492) at org.eclipse.m2e.core.internal.project.registry.ProjectRegistryManager$3.call(ProjectRegistryManager.java:1) at org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.executeBare(MavenExecutionContext.java:177) at org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.execute(MavenExecutionContext.java:151) at org.eclipse.m2e.core.internal.project.registry.ProjectRegistryManager.refresh(ProjectRegistryManager.java:496) at org.eclipse.m2e.core.internal.project.registry.ProjectRegistryManager.refresh(ProjectRegistryManager.java:351) at org.eclipse.m2e.core.internal.project.registry.ProjectRegistryManager.refresh(ProjectRegistryManager.java:298) at org.eclipse.m2e.core.internal.project.ProjectConfigurationManager.updateProjectConfiguration0(ProjectConfigurationManager.java:405) at org.eclipse.m2e.core.internal.project.ProjectConfigurationManager$2.call(ProjectConfigurationManager.java:352) at org.eclipse.m2e.core.internal.project.ProjectConfigurationManager$2.call(ProjectConfigurationManager.java:1) at org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.executeBare(MavenExecutionContext.java:177) at org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.execute(MavenExecutionContext.java:151) at org.eclipse.m2e.core.internal.embedder.MavenExecutionContext.execute(MavenExecutionContext.java:99) at org.eclipse.m2e.core.internal.embedder.MavenImpl.execute(MavenImpl.java:1351) at org.eclipse.m2e.core.internal.project.ProjectConfigurationManager.updateProjectConfiguration(ProjectConfigurationManager.java:349) at org.eclipse.m2e.core.ui.internal.UpdateMavenProjectJob.runInWorkspace(UpdateMavenProjectJob.java:77) at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:39) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:56) Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.flink:flink-scala_2.11:pom:1.6.0 from/to central (https://repo.maven.apache.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223) at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:267) ... 33 more Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact org.apache.flink:flink-scala_2.11:pom:1.6.0 from/to central (https://repo.maven.apache.org/maven2): java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at io.takari.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:895) at io.takari.aether.connector.AetherRepositoryConnector$2.wrap(AetherRepositoryConnector.java:1) at io.takari.aether.connector.AetherRepositoryConnector$GetTask.flush(AetherRepositoryConnector.java:673) at io.takari.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:310) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:520) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:421) ... 36 more Caused by: javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1964) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1921) at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1904) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1420) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:268) at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:238) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:149) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:192) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:121) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:100) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) at okhttp3.RealCall.execute(RealCall.java:69) at io.takari.aether.okhttp.OkHttpAetherClient.execute(OkHttpAetherClient.java:207) at io.takari.aether.okhttp.OkHttpAetherClient.get(OkHttpAetherClient.java:153) at io.takari.aether.connector.AetherRepositoryConnector$GetTask.resumableGet(AetherRepositoryConnector.java:600) at io.takari.aether.connector.AetherRepositoryConnector$GetTask.run(AetherRepositoryConnector.java:453) at io.takari.aether.connector.AetherRepositoryConnector.get(AetherRepositoryConnector.java:304) ... 38 more Caused by: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:91) at sun.security.validator.Validator.getInstance(Validator.java:181) at sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:312) at sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:171) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:184) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1596) at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) at sun.security.ssl.Handshaker.process_record(Handshaker.java:987) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ... 63 more Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200) at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120) at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104) at sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:89) ... 75 more 编辑:只是一些语法和清晰度

flink小助手

在Eclipse中创建Flink项目

我正在尝试用eclipse创建一个Flink Maven项目并运行一些自写代码文件。我使用了这个教程:https://ci.apache.org/projects/flink/flink-docs-release-1.6/quickstart/java_api_quickstart.html 我完成了本教程的所有步骤。如果我尝试运行示例文件,则会出现以下错误消息: 此外,如果我正在尝试生成一个新的Java类 - 错误 源文件夹不是java项目。 我的目标是创建自己的Flink应用程序并在IDE环境中测试它们。

flink小助手

Flink:找不到合适的方法

我正在尝试在Flink的DataStream上应用每个窗口的功能。以下是我的代码 DataStream> data = ...DataStream> freqCityChangeTransactions = data .keyBy(0) .timeWindow(Time.seconds(5)) .process(new MyProcessWindowFunction()); 以下是我对MyProcessWindowFunction的实现 public static class MyProcessWindowFunctionextends ProcessWindowFunction, Tuple2, String, TimeWindow> { public void process(String key, Context context, Iterable<Tuple2<String, String>> input, Collector<Tuple2<String, String>> out) { // Do something ... } }但是,当我尝试通过maven编译上面的代码时,我收到以下错误 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project flink-examples: Compilation failure[ERROR] /Users/furqan/Workspace/flink/src/main/java/com/baig/bank/Bank.java:[120,13] no suitable method found for process(com.baig.Bank.MyProcessWindowFunction)[ERROR] method org.apache.flink.streaming.api.datastream.WindowedStream.process(org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction,R,org.apache.flink.api.java.tuple.Tuple,org.apache.flink.streaming.api.windowing.windows.TimeWindow>) is not applicableERROR RERROR)[ERROR] method org.apache.flink.streaming.api.datastream.WindowedStream.process(org.apache.flink.streaming.api.functions.windowing.ProcessWindowFunction,R,org.apache.flink.api.java.tuple.Tuple,org.apache.flink.streaming.api.windowing.windows.TimeWindow>,org.apache.flink.api.common.typeinfo.TypeInformation) is not applicableERROR RERROR)我正在使用Apache Flink 1.5.1版并在Mac上使用maven3编译Java代码。

社区小助手

Spark sql和hive导致“错误:无法找到或加载主类”

我使用Scala开发了一个使用hive进行通信的spark应用程序。它在Intellij的想法上工作得很好。但是当我构建一个具有所有依赖性的jar文件时,我收到错误 错误:无法找到或加载主类 经过进一步调试后,我发现当我包含spark SQL或hive时,我收到错误。这是一个maven项目。请查看我的pom.xml 4.0.0 spkspk1.0-SNAPSHOT <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.11</artifactId> <version>2.0.2</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.11</artifactId> <version>2.0.2</version> </dependency> <!--<dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-hive_2.11</artifactId> <version>2.0.2</version> </dependency>--> <!--<dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>1.2.2</version> </dependency>--> <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.11</artifactId> <version>2.0.1</version> </dependency> <!-- https://mvnrepository.com/artifact/com.101tec/zkclient --> <dependency> <groupId>com.101tec</groupId> <artifactId>zkclient</artifactId> <version>0.11</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients --> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> <version>0.10.0.0</version> </dependency> <sourceDirectory>src/main/scala</sourceDirectory> <plugins> <plugin> <groupId>org.scala-tools</groupId> <artifactId>maven-scala-plugin</artifactId> <version>2.11</version> <executions> <execution> <goals> <goal>compile</goal> <goal>testCompile</goal> </goals> <configuration> <args> <!--<arg>-make:transitive</arg>--> <arg>-dependencyfile</arg> <arg>${project.build.directory}/.scala_dependencies</arg> </args> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.11</version> <configuration> <useFile>false</useFile> <disableXmlReport>true</disableXmlReport> <!-- If you have classpath issue like NoDefClassError,... --> <!-- useManifestOnlyJar>false</useManifestOnlyJar --> <includes> <include>**/*Test.*</include> <include>**/*Suite.*</include> </includes> </configuration> </plugin> </plugins>