个人安装使用sbt流程及遇到的问题
安装流程
1、官网下载sbt , https://www.scala-sbt.org/download.html
2、测试sbt是否可用
//进入sbt目录
cd /opt/apps/sbt
./sbt sbtVison
显示无法找到,查看sbt目录发现sbt文件在bin目录下,改为
./bin/sbt sbtVersion
3、回到主目录,创建sparkapp文件夹作为应用程序根目录
cd ~
mkdir ./sparkapp
mkdir -p ./sparkapp/src/main/scala
进入sparkapp目录下查看创建情况
cd sparkapp
find .
.
./src
./src/main
./src/main/scala
4、进入/src/main/scala创建SipmleApp.scala
/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
object SimpleApp{
def main(args: Array[String]){
val logFile = "hdfs://hadoop0:9000/data/word/in/word"
val conf = new SparkConf().setAppName("Simple Application")
val sc = new SparkContext(conf)
val logData = sc.textFile(logFile,2).cache()
val numAs = logData.filter(_.contains("a")).count()
val numBs = logData.filter(_.contains("b")).count()
println("Lines with a: %s,Lines with b: %s".format(numAs,numBs))
}
}
5、使用sbt对Scala程序进行打包
回到sparkapp,创建simple.sbt
name := "Simple Project"
version := "1.0"
scalaVersion := "2.12.15"
libraryDependencies += "org.apache.spark" %% "spark-core" %"3.3.0"
输入命令/opt/apps/sbt/bin/sbt package
,编译成功就会出现如下信息
[info] Loading project definition from /root/sparkapp/project
[info] Loading settings for project sparkapp from simple.sbt ...
[info] Set current project to Simple Project (in build file:/root/sparkapp/)
[success] Total time: 1 s, completed Oct 18, 2022 12:10:02 AM
遇到的问题
1、./bin/sbt sbtVersion
命令后会开始下载
出现Getting org.Scala-sbt 1.3.8 (this may take some times)...
后很长时间不动
在sbt目录下创建repositories文件,并添加
[repositories]
local
huaweicloud-maven: https://repo.huaweicloud.com/repository/maven/
maven-central: https://repo1.maven.org/maven2/
sbt-plugin-repo: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases, [organization]/[module]/(scala_[scalaVersion]/)(sbt_[sbtVersion]/)[revision]/[type]s/[artifact](-[classifier]).[ext]
2、下载scala-2.12.10报错 error: not found: value scalaversion scalaversion := "2.12.15"
打开simple.sbt,去掉scalaVersion := "2.12.15"
一行
3、编译时报错
在spark官网 https://spark.apache.org/docs/latest/streaming-programming-guide.html#a-quick-example 找到
文章来源:https://www.toymoban.com/news/detail-500844.html
将simple.sbt中的libraryDependencies += "org.apache.spark" %% "spark-core" %"3.3.0"
改为libraryDependencies += "org.apache.spark" % "spark-streaming_2.12" % "3.3.0" % "provided"
文章来源地址https://www.toymoban.com/news/detail-500844.html
到了这里,关于个人安装配置sbt流程及遇到的问题的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!