Used to be in the Eclipse new project, through Maven import Spark depend on the package, now need to modify the Spark source,
From the official website to download the Source Code, the modified don't know how to use in the Eclipse project, hope to get a great god give directions!

CodePudding user response:

Development and debugging environment under the Eclipse configuration

The section using all the tools are: Windows 7 + + Scala Eclipse Java EE 4.4.2 2.10.4 + Sbt 0.13.8 + Maven3.3.3, test version of the Spark for 1.4.0,

1. Configure the IDE:

Select a menu item Help - & gt; Install new software, add a site http://download.scala-ide.org/sdk/lithium/e44/scala211/stable/site, choose to Install Scala IDE for Eclipse and Scala IDE Plugins,


For a standard version of the Eclipse also need to install a separate Maven plug-ins,

For the sake of simple configuration, also can use Scala official offer has been packaged all depend on good Scala IDE,

In particular, due to the Project itself has some mistakes, please temporarily shut down the Project - & gt; Build the function of Automatically to save time,

2. Download the Spark source:

Create an empty directory, perform the following statement: git clone https://github.com/apache/spark.git

In addition to using the git commands, also can download it from the Spark of making pages packed source code,

3. The source code into Eclipse project:

Into the root directory of the source code, perform the following statement: SBT eclipse, SBT during execution will download the Spark needed to all the jars, so this step will take a very long time, some of the jar packages need to use methods such as network proxy to download,


4. Import the project into Eclipse:

Select a menu item File - & gt; The Import, and select the General - & gt; Existing Projects into Workspace, root path choice of the source code path of the project, the import all items (total of 25),

5. Modify the Scala version:

Enter the Preference - & gt; Scala - & gt; Installations, adding machine install Scala 2.10.4 select the lib directory), because this version of the Spark (1.4.0) is in Scala 2.10.4 environment, need to modify the project in Eclipse using the Scala version, the method is: all project, right-click the Scala - & gt; Set the Scala Installation and select the corresponding Scala version,


6. As the old - deps project added Scala Library:

Right-click to choose old - deps project, select the Scala - & gt; Add the Scala Library to Build Path,

7. Maven install to generate spark - streaming - the flume - sink required classes:

Source code in the root directory of the scalastyle - config. XML files are copied to the spark - streaming - the flume - sink project root directory, and then open the project in Eclipse, right-click pom. The XML file, select Run as - & gt; Maven install,


8. Modify the spark to SQL and spark to hive packet error:

As a result of the source code package Settings there is an error, for this class files need to be moved to the correct package


For spark - SQL project, select the SRC/test/Java test.org.apache.spark.sql and test.org.apache.spark.sql.sources in all the classes in the package, right-click to choose Refactor - & gt; Move, Move to org. Apache. Spark. SQL and org. Apache. Spark. SQL. Sources packets,

To spark - hive projects are selected in the SRC/test/Java test.org.apache.spark.sql.hive and test.org.apache.spark.sql.hive.execution package all the classes, in the movement to org. Apache. Spark. SQL. Hive and org. Apache. The spark. SQL. Hive. Execution package,

9. Compile all projects:

Open the Project - & gt; Build Automatically function, wait for all project successful compilation,

10. Check whether the installation is successful:

Will the core project of SRC - & gt; The main - & gt; Resources - & gt; Org copy folder to the examples in the project target - & gt; Scala - 2.10 - & gt; In the classes, and then execute examples project org. Apache. Spark. Examples. SparkPi procedure, and set its JVM parameters for - Dspark. Master=local