Background

Project Alice generates Java source code, stores it in sources.jar, then uploads it to a Maven repository. Project Bob pulls sources.jar down and needs to use it when compiling. Bob does not know that Alice exists, only where to find sources.jar.

Versions: JDK 11, Gradle 7.3.1, IntelliJ IDEA 2021.3.1

Problem

Making gradle (and IntelliJ's IDEA) build using source files embedded in a JAR file. To be clear, the JAR file contents resemble:

$ jar -tvf sources.jar 
     0 Thu Feb 03 08:38:56 PST 2022 META-INF/
    52 Thu Feb 03 08:38:56 PST 2022 META-INF/MANIFEST.MF
     0 Thu Feb 03 08:38:30 PST 2022 com/
     0 Thu Feb 03 08:38:32 PST 2022 com/domain/
     0 Thu Feb 03 08:38:30 PST 2022 com/domain/package/
   938 Thu Feb 03 08:38:32 PST 2022 com/domain/package/SourceCode.java

Solutions that extract the .java files from the .jar file introduce knock-on effects we'd like to avoid, including:

• Editable. Extracted source files can be edited in the IDE. We'd like them to be read-only. We could add a task that sets the files read-only, but that feels like solving the wrong problem (added complexity).
• Synchronization. When a newly generated sources.jar is pulled down, we'll have to delete the extraction directory to remove any stale .java files that were preserved. If there was a way to avoid extraction, then the act of pulling down the new sources.jar file would ensure correctness (no added complexity). By unarchiving the .java files, it's possible to enter an inconsistent state:

$ jar -tvf sources.jar | grep java$ | wc -l
61
$ find src/gen -name "*java" | wc -l
65

If there was a way to treat sources.jar as a source directory without extracting the files, these knock-on effects disappear.

Attempts

A number of approaches have failed.

sourceSets

Changing sourceSets doesn't work:

sourceSets.main.java.srcDirs  = "jar:${projectDir}/sources.jar!/"

The error is:

Cannot convert URL 'jar:/home/user/dev/project/sources.jar!/' to a file.

Using a zipTree with sourceSets doesn't work, although the error message is telling:

sourceSets.main.java.srcDirs  = zipTree(file: "${projectDir}/sources.jar")

Error:

Cannot convert the provided notation to a File or URI.

The following types/formats are supported:

• A URI or URL instance.

This was expected. What was unexpected was that URL instances are allowed, but seemingly not if embedded within a JAR file.

The following allows building Bob, but the IDE is unable to find SourceCode.java:

sourceSets.main.java.srcDirs  = zipTree("${projectDir}/sources.jar").matching {
  include "com"
}

build task

Modifying the build task to extract the generated code first partially works:

task codeGen {
  copy {
    from( zipTree( "${projectDir}/sources.jar" ) )
    into( "${buildDir}/src/gen/java" )
  }

  sourceSets.main.java.srcDirs  = ["${buildDir}/src/gen/java"]
}

build { doFirst { codeGen } }

The issue is that removing the build directory then prevents static compiles (because IDEA cannot find the generated source files). In any case, we don't want to extract the source files because of all the knock-on problems.

compile task

The following snippet also does not compile:

tasks.withType(JavaCompile) {
  source = zipTree(file: "${projectDir}/sources.jar")
}

Not updating sourceSets means that the IDE cannot discover the source files.

sync task

We could extract the files into the main source directory, instead, such as:

def syncTask = task sync(type: Sync) {
    from zipTree("${projectDir}/sources.jar")
    into "${projectDir}/src/gen/java"

    preserve {
        include 'com/**'
        exclude 'META-INF/**'
    }
}

sourceSets.main.java.srcDir(syncTask)

While that addresses the clean issue, we’re left with the original problems that we’d like to avoid.

Content Root

Setting the Content Root and marking the Source Folder from within IntelliJ IDEA works. The IDE updates .idea/misc.xml to include:

  <component name="FrameworkDetectionExcludesConfiguration">
    <file type="web" url="file://$PROJECT_DIR$" />
    <file type="web" url="jar://$PROJECT_DIR$/project/sources.jar!/" />
  </component>

In theory, the idea plugin may have the ability to set this value.

Question

How would you instruct Gradle to reference and build using source files that are stored in an external Java archive file when compiling a project (without extracting the archive) such that IDEA can statically resolve the source files, as well?

CodePudding user response：

Working Solution

I first believed it wasn’t possible to use a JAR file containing uncompiled Java code as additional sources in IntelliJ. After a few tries I could eventually configure it in the UI, though, thanks to the pointer from your “Content Root” section. A bit of fiddling with the IDEA plugin later, I could finally come up with a fully working solution:

plugins {
    id 'java'
    id 'idea'
}

tasks.withType(JavaCompile) {
    source(zipTree("${projectDir}/sources.jar"))
}

idea.module.iml {
    withXml {
        def baseUrl = 'jar://$MODULE_DIR$/sources.jar!/'
        def component = it.asNode().component[0]
        def jarContent = component.appendNode('content', [url: baseUrl])
        jarContent.appendNode('sourceFolder', [
                url: "${baseUrl}com",
                isTestSource: false,
            ])
    }
}

Before opening the project in IntelliJ, you’ll have to run ./gradlew idea and then open the generated .ipr file with IntelliJ. If IntelliJ says “Gradle build scripts found”, then don’t attempt to load the Gradle project but instead “Skip” this. You now have an IntelliJ project which is generated by Gradle but which is still independent of Gradle – and hence you can safely call ./gradlew clean without affecting IntelliJ.

Other Thoughts

Here’s another approach of configuring the compiler from Gradle, using the (rarely used) -sourcepath option of javac:

tasks.withType(JavaCompile) {
    options.sourcepath = files("${projectDir}/sources.jar")
}

I personally would still prefer an approach where I don’t have to let Gradle generate an IntelliJ project but instead to let IntelliJ work with Gradle. However, that’ll require extracting the JAR file. A good Gradle solution for this would be the following:

plugins {
    id 'java'
}

def unzipAlice = tasks.register('unzipAlice', Sync) {
    from(zipTree("${projectDir}/sources.jar"))
    into(temporaryDir)
}

sourceSets.main.java.srcDir(unzipAlice)

This solution at least solves the “synchronization” problem mentioned in your question: the Sync task makes sure that no updates to sources.jar are ignored (including deletions).

About the “editable” problem, I wonder why you don’t trust your developers here? They could produce all kind of nonsense in the rest of the code anyway. Or even if the JAR isn’t extracted, they could still replace the files in the JAR. With this extraction solution, IntelliJ at least warns if the sources are edited:

Generated source files should not be edited. The changes will be lost when sources are regenerated.

Ok, if you run ./gradlew clean then IntelliJ will indeed not find the sources anymore. But this can easily be fixed by either calling “Build” → “Build Project” in the UI or by running “./gradlew unzipAlice”. If the “./gradlew clean” issue should really be a dealbreaker for the extracting approach, then you could still consider extracting the sources outside of the build directory …

CodePudding user response：

We decided that extracting the jar file would be the best approach after all:

apply plugin: 'idea'

final GENERATED_JAR = "${projectDir}/sources.jar"
final GENERATED_DIR = "${buildDir}/generated/sources"

task extractSources(type: Sync) {
  from zipTree(GENERATED_JAR)
  into GENERATED_DIR
}

sourceSets.main.java.srcDir extractSources
clean.finalizedBy extractSources

idea.module.generatedSourceDirs  = file(GENERATED_DIR)

This:

• Retains the generated sources after clean
• Issues a warning in the IDE when modifying generated sources
• Couples the build process to extracting source files
• Reuses the build/generated/sources path
• Keeps .jar file and .java files in synchronization

In effect, the following behaviours work as expected:

• ./gradlew clean -- leaves the extracted .java files intact, effectively
• ./gradlew build -- re-synchronizes .java files with .jar contents