Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kotlin] Kotlin support #3808

Merged
merged 42 commits into from
Jun 30, 2022
Merged

[kotlin] Kotlin support #3808

merged 42 commits into from
Jun 30, 2022

Conversation

adangel
Copy link
Member

@adangel adangel commented Feb 25, 2022

Describe the PR

Follow-up on #3505

Related issues

Related work

Ready?

jborgers and others added 22 commits September 7, 2021 16:17
…d language" except $. Generate your Parser did not succeed because combining lexer and parser grammars has issues.
…ammar; and move PmdKotlinParser.java to the right place
…lasses.g4 file, added <libDirectory> to maven antlr plugin to make it work
…-SNAPSHOT in sonatype snapshots repo"

This reverts commit 051da73.
@adangel adangel added this to the 7.0.0 milestone Feb 25, 2022
Copy link
Member

@oowekyala oowekyala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot Andreas for cleaning this up!

docs/_data/xpath_funs.yml Outdated Show resolved Hide resolved
docs/_data/xpath_funs.yml Outdated Show resolved Hide resolved
Comment on lines 4 to 29
- PackageHeader
| - T-PACKAGE
| - Identifier
| | - SimpleIdentifier
| | | - T-Identifier
| | - T-DOT
| | - SimpleIdentifier
| | | - T-Identifier
| | - T-DOT
| | - SimpleIdentifier
| | | - T-Identifier
| | - T-DOT
| | - SimpleIdentifier
| | | - T-Identifier
| | - T-DOT
| | - SimpleIdentifier
| | | - T-Identifier
| | - T-DOT
| | - SimpleIdentifier
| | | - T-Identifier
| | - T-DOT
| | - SimpleIdentifier
| | - T-Identifier
| - Semi
| - T-NL
| - T-NL
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's how package net.sourceforge.pmd.lang.kotlin.ast.testdata is currently parsed. That's unfortunately very verbose and that's how the antlr grammar is written. That's why I want to mark this language as experimental, so that we can maybe simplify the AST somehow and don't worry about compatibility.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. We use @joinTokenText in rules to address the AST verbosity, however, it is deprecated because of bad performance. Would be nice to have an alternative which keeps working with the improved AST.

pom.xml Show resolved Hide resolved
pmd-kotlin/pom.xml Outdated Show resolved Hide resolved
@pmd-test
Copy link

pmd-test commented Feb 25, 2022

2 Messages
📖 Compared to pmd/7.0.x:
This changeset changes 36 violations,
introduces 28 new violations, 0 new errors and 0 new configuration errors,
removes 30 violations, 0 errors and 0 configuration errors.
Full report
📖 Compared to master:
This changeset changes 57095 violations,
introduces 34423 new violations, 2 new errors and 0 new configuration errors,
removes 138133 violations, 25 errors and 6 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset changes 29 violations,
introduces 20 new violations, 0 new errors and 0 new configuration errors,
removes 23 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset changes 57094 violations,
introduces 34425 new violations, 2 new errors and 0 new configuration errors,
removes 138134 violations, 25 errors and 6 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset changes 22 violations,
introduces 24 new violations, 0 new errors and 0 new configuration errors,
removes 20 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset changes 57348 violations,
introduces 34626 new violations, 7 new errors and 0 new configuration errors,
removes 138168 violations, 25 errors and 3 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset changes 23 violations,
introduces 23 new violations, 0 new errors and 0 new configuration errors,
removes 20 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset changes 57267 violations,
introduces 34624 new violations, 7 new errors and 0 new configuration errors,
removes 138165 violations, 25 errors and 3 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset changes 22 violations,
introduces 21 new violations, 0 new errors and 0 new configuration errors,
removes 28 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset changes 57267 violations,
introduces 34614 new violations, 7 new errors and 0 new configuration errors,
removes 138165 violations, 25 errors and 3 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset changes 27 violations,
introduces 24 new violations, 0 new errors and 0 new configuration errors,
removes 18 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset changes 57349 violations,
introduces 34628 new violations, 7 new errors and 0 new configuration errors,
removes 138167 violations, 25 errors and 3 configuration errors.
Full report

Generated by 🚫 Danger

@adangel adangel mentioned this pull request Feb 25, 2022
4 tasks
@jborgers
Copy link
Contributor

jborgers commented Mar 7, 2022

@adangel @oowekyala A question, not sure this is the right place.
PMD7 is incompatible with PMD6, the AST is different. This means it is quite some work to migrate rule definitions to PMD7. We want to analyze Kotlin code by a central SonarQube server as well as Java code. To enable this at the same time and not having to wait for migration of all our 98 custom Java rules, we need to run PMD7-Kotlin as well as PMD6-Java in one SonarQube server. To make this work, we chose to have a sonar-pmd-plugin and a sonar-pmd7-plugin running in Sonar, and have separate identifiers (pmd vs pmd7) for repository keys, rule keys and such. It works, however, I am not sure this is the best approach. What do you think? Any comments, suggestions? Ways to ease the migration?

@adangel
Copy link
Member Author

adangel commented Mar 11, 2022

@adangel @oowekyala A question, not sure this is the right place. PMD7 is incompatible with PMD6, the AST is different. This means it is quite some work to migrate rule definitions to PMD7. We want to analyze Kotlin code by a central SonarQube server as well as Java code. To enable this at the same time and not having to wait for migration of all our 98 custom Java rules, we need to run PMD7-Kotlin as well as PMD6-Java in one SonarQube server. To make this work, we chose to have a sonar-pmd-plugin and a sonar-pmd7-plugin running in Sonar, and have separate identifiers (pmd vs pmd7) for repository keys, rule keys and such. It works, however, I am not sure this is the best approach. What do you think? Any comments, suggestions? Ways to ease the migration?

I think, you can't avoid updating your custom rules at some point to fully migrate to pmd7.

I don't know, how the integration of sonar into the build works. I wonder, whether you really need two separate versions of the sonar-pmd-plugin, maybe two execution instances would work? E.g. the maven-pmd-plugin can also work with pmd7, so the API to call pmd is (at least for now) still compatible between pmd6 and pmd7. That would be "just" a dependency update. But in your case, it's a bit more, because sonar-pmd-plugin knows only java and not kotlin.

Ok, I looked at how the sonar-pmd-plugin is installed - it's a single jar file including all the dependencies, so you have no chance to configure the dependencies at runtime as you need (like with maven).

So, the easiest way seems to be indeed what you already do: complete separate plugin for pmd7.
In theory, it could be possible to merge the two plugins, but you would need to use separate class loaders... and package the dependencies in a different way (as you need both pmd6 and pmd7 dependencies, but not both on the default plugin's classpath).

@jborgers
Copy link
Contributor

jborgers commented Mar 15, 2022

Thanks for your answer. Right, good to know we are on the right track. We should be able to build both versions of the plugin, so not as separate branch. Choice to make is between 1. two separate directory structures and duplication of code and 2. somehow parameterise to manage the differences, with more complexity.

@stokpop
Copy link
Contributor

stokpop commented Jun 10, 2022

Hi @adangel @oowekyala, me and @jborgers are working on the sonar-pmd-plugin to get it working with pmd7 with Kotlin added. Currently we need to locally build https://github.com/adangel/pmd/tree/kotlin-poc.

Is it possible to merge kotlin-poc in the pmd/7.0.x branch so we can use the pmd 7.0.0-SNAPSHOT release from Sonatype snapshots instead? What needs to be done to get this branch merged and can we help with that? Thanks.

@stokpop
Copy link
Contributor

stokpop commented Jun 21, 2022

Hi @adangel and @oowekyala, we run into an issue creating the sonar-pmd plugin with pmd7. When running mvn sonar:sonar the maven process would just exit with status 1 without logging what was wrong. Remote debugging showed: java.lang.NoSuchMethodError: 'boolean org.slf4j.Logger.isEnabledForLevel(org.slf4j.event.Level)' which is called from pmd7 code, via:

realm =    plugin>org.codehaus.mojo:sonar-maven-plugin:3.9.1.2184
strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy

The pmd-7.0.0-SNAPSHOT.jar makes use of slf4j 2.0.0-alpha6 as seen by mvn dependency:tree

org.sonarsource.pmd:sonar-pmd7-plugin:sonar-plugin:3.4.1-SNAPSHOT
 - net.sourceforge.pmd:pmd-java:jar:7.0.0-SNAPSHOT:compile
|   - net.sourceforge.pmd:pmd-core:jar:7.0.0-SNAPSHOT:compile
|  |   - org.slf4j:slf4j-api:jar:2.0.0-alpha6:compile

This is not compatible with Sonar, that provides slf4j to its plugins. We can exclude 2.0.0-alpha6, but 1.7.x is provided.

We could solve this by making a local build of pmd-7.0.0-SNAPSHOT.jar with a dep on slfj4j 1.7.36 and rewriting pmd7 code that uses slf4j 2.0.0 specific methods.

Another way to solve this, it seems, is to configure the sonar-packaging-maven-plugin to use <useChildFirstClassLoader>true</useChildFirstClassLoader>. Alas this property is deprecated and not future proof and might introduce other classloader issues.

From the sonar plugin instructions:

Under: Third-party Libraries
It says: "Note that since version 5.2, the SonarQube API does not bring transitive dependencies, except SLF4J."
Under: Logging
It says: "As an exception, plugins must not package logging libraries. Dependencies like SLF4J or log4j must be declared with scope "provided"."

We wanted to bring this to your attention: using slf4j 2.0.0 might not work everywhere when pmd is used as jar.

Also, what would you recommend in this case?

@adangel adangel mentioned this pull request Jun 24, 2022
4 tasks
@adangel
Copy link
Member Author

adangel commented Jun 24, 2022

@stokpop I've created #4023 for downgrading slf4j. I don't think there is a different way - there can only be one logging framework active.
Excluding slf4j2 is not an option - you would run into the same NoSuchMethodErrors.
Using the useChildFirstClassloader only works, if you add additionally a slf4j2-comptible logging backend - which means, you would have 2 logging backends active in sonar potentially writing to the same stream/file...

@adangel adangel linked an issue Jun 30, 2022 that may be closed by this pull request
@adangel adangel merged commit 05bb086 into pmd:pmd/7.0.x Jun 30, 2022
@adangel adangel deleted the kotlin-poc branch June 30, 2022 16:24
@stokpop
Copy link
Contributor

stokpop commented Jul 5, 2022

@adangel Thanks for the slf4j downgrade, I tested your branch in the adangel repo and it works fine. Also the Kotlin merge seems to work. I also run into a concurrency issue, I reported here: #4035

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:new-language Proposal to add a new PMD or CPD language to the main distribution
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[kotlin] Add support for Kotlin
5 participants