-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The Kotlin and Groovy DSLs should have similar performance characteristics #15886
Comments
thanks for the benchmark @eskatos Any issues filed on the youtrack issue tracker against Kotlin compiler related to kotlin dsl performance? if so please link the associated issues. |
random thoughts from a lambda Kotlin dev: It could be a useful addition to add a variant for kotlin and groovy on this benchmark: org.gradle.vfs.watch=true
kapt.use.worker.api=true
kapt.include.compile.classpath=false
org.gradle.parallel=true
org.gradle.caching=true (btw its unclear whether kapt.include.compile.classpath=false is still useful in 6.8 given compiler avoidance) Also worth exploring would be to benchmark with the new compiler backend and the new compiler frontend finally setting a continuous benchmark (in CI) of this would be useful for showing progress and for catching regressions! note1: I cannot see the linked benchmark results,
https://docs.gradle.org/nightly/userguide/performance.html#compiler_daemon note3: minor: I wonder whether using the Kotlin BOM incurs a slowdown vs not using it |
maybe worth adding one more case, change build.gradle file (not src), I think it even more common comparing to buildSrc |
Guest access to our CI is currently temporarily disabled, it'll be enabled back soon. @kaushalyap there's already a link to https://youtrack.jetbrains.com/issue/KT-24668. I'm not aware of other Youtrack issues related to this. @LifeIsStrange this issue is about the Gradle Kotlin DSL performance, not the Kotlin Gradle Plugin performance in general. For instance KAPT is out of the picture. Also, the embedded Kotlin compiler used internally by the Gradle Kotlin DSL isn't ready to be moved to the new compiler infrastructure. The numbers above represent the default Gradle user experience which is a good thing. @gildor yes this use case is missing from the set above and I agree it's very common. We started with the |
@eskatos I know about that issue. Hope Kotlin team close that issue in the first half of this year with the movement to new compiler as mentioned in the roadmap, so android devs can move to kotlin dsl for android projects without increased build time. |
@kaushalyap This is the new backend of the compiler, it has nothing to do with compilation speed. You probably talk about the new frontend of the compiler, which promises up to 2 times faster compilation, but it will not be ready in 6 months, in the best case it would be some experimental version available this year |
I wonder if this is still an issue. Gradle 7 has been out for a while and it boosts massive performance improvements for Kotlin DSL script compilation. An updated benchmark would be very appreciated :). |
Those benchmarks are automated and constantly run against Gradle as it evolves. You can browse the results on our CI, e.g. here https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger/46861573?buildTab=report_project55641_Performance Here I captured the current state:
Things indeed got better since Gradle 7, especially the non-abi change scenarios. The performance isn't similar though. |
Could you explain what ABI / Nop-ABI means 🙈 |
@rowi1de sure thing! ABI stands for Application Binary Interface. An ABI change is a change that modifies the ABI, iow the "public api" that downstream consumers compile against. All consumers need to be recompiled. A non-ABI change is a change that does not modify the ABI, iow only touches the "private implementation". Consumers compilation can be skipped. |
@blundell that individual run got garbage collected, here you can see the list of runs https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger?branch=master&buildTypeTab=overview&mode=builds |
I think I've done this right :D Gradle 7.4/7.5 numbers
Info from 11th Feb 2022 CI run For example, from the performance tab, I looked up the buildSrc abi change and took the first row, here. |
@blundell , could you please explain the difference calculations? In the previous results: 21s VS 4.3s - 4.8x faster. I get it. |
@eugene-krivobokov Sorry about that, wrong equation. Updated to calculate as a percentage. |
does it private? seems unable to open with my team city account |
Is there any sort of break down on what goes in "first use" and why it is so much slower? |
@liutikas I guess "first use" is slower with Kotlin DSL as that needs to be compiled. |
build.gradle.kts configuration files was replaced by build.gradle # Ref: gradle/gradle#15886 (comment)
* build: speed up build time (part 1) # Ref: https://habr.com/ru/company/yandex/blog/678786/ * build: speed up build time (part 2) build.gradle.kts configuration files was replaced by build.gradle # Ref: gradle/gradle#15886 (comment)
Here are numbers updated for Gradle 8.0
This is not looking great. The upgrade to Kotlin 1.8 in Gradle 8.0 caused a regression in Kotlin DSL script compilation. Also those scenarios are not the best at representing reality as they don't even use the The gap is not just the Kotlin compiler performance change. Groovy got faster. There are also some changes due to other areas of Gradle at play. But there definitely was a performance regression with 1.8 in our isolated tests. We are discussing this with the compiler team and will try to mitigate. |
@eskatos could you please check gradle 8.1 performance |
8.2 is out with Kotlin DSL is Now the Default for New Gradle Builds. I really would like to know whether there are any performance improvements related to previous versions. Is there any chance to do some testing against the latest Gradle release? Thanks. |
I haven’t run any benchmark on the latest release but looking at the Gradle’s public roadmap, I think we need to wait Gradle 8.4 at least for first-use performance improvements |
Since Groovy is OBVIOUSLY faster and a more mature lang, any chance there will be a reversion back? NOTE: I notice the thumbs down so am confused as people want speed but also want 'Kotlin'??? You can't complain about speeds and then reject the one thing that gave you that speed. This is a 'cake and eat it' situation folks |
thanks for sharing the insights, Since Google is the default kotlin for builds, curious to know about performance with AGP 8.1 onwards before I make a move & update the whole project which has at least 50 modules & Build performance is a higher priority for big team, i tried in older version which was very slow |
It would be nice to see updated numbers for Gradle 8.7 given the changes in #27271. |
Here are numbers updated for Gradle 8.7
Things have not changed much overall since 8.0. The gap between the two DSLs closed a tiny bit. |
Lol. soooo... now would be the time a sensible person would say 'let's revert back to #groovy. This isn't working out'. |
I wonder if the new (not yet released) Kotlin K2 compiler is going to make things better here. |
Thats a fun game to play; bet on future compiler against future compiler; this is actually how we got here. Instead, why not just use PRESENT tools UNTIL other tools meet requirements. GASP RADICAL IDEA! |
I missed some news? |
Oh didn't you hear? https://android-developers.googleblog.com/2023/04/kotlin-dsl-is-now-default-for-new-gradle-builds.html#:~:text=gradle.,Compose%2C%20and%20now%20build%20scripts! |
This one I've heard. My pretty big(couple of mln lines of code, and thousands of Gradle modules) project also suffers from some Gradle performance issues, and it's actually still on groovy scripts exactly because of the data from this thread and personal benchmarks. However until groovy is on par with kotlin-dsl how is this even a problem? Please explain, I probably missing something |
Uh huh... BUT you are using 'the Microsoft Monopoly argument' : 'IE is the default browser, not the required browser' You converted to a language that is less scalable to please a BILLION DOLLAR COMPANY. The end user has to KNOW to go into THIS ISSUE to see the benchmarks and understand the differences/deficiencies The will not know to do this. So you are making all decisions for the end user. Thus most will use the default... much like most used the DEFAULT BROWSER in Microsoft (and why they got sued for a monopoly) Here is where you blow off my response as meaningless. |
I agree that making it the default is a questionable decision. Probably it's working "good enough" for most of the project around here, but it definitely not the best choice for 1k+ modules monsters right now. I also have this thing to add: |
But the ORIGINAL reasoning for the push was 'KOTLIN would be FASTER'. it wasn't. It isn't. It's been proven. Now we sit here and try to compromise on the reasons for the decision after the fact like a bad management decision rather than engineers accepting that this is (and was) a bad engineering decision. So the question is: is this a management decision or an engineering decision? Because thats what ends this discussion :) A management discussion is just so they can please their corporate benefactors. An engineering discussion is to have a better product. So that is the real question. Is this going to be an engineering conversation or a management one? |
What's the source of that? The story was always around improving the editing experience by using a type safe language that can offer proper auto complete and other IDE features. https://blog.gradle.org/kotlin-meets-gradle |
You are showing articles for adding Kotlin to Gradle... not for making it the DEFAULT; thats what we are discussing. But you are 100% right. No one EVER said speed was a reason to adopt Kotlin https://blog.jetbrains.com/kotlin/2023/04/kotlin-dsl-is-the-default-for-new-gradle-builds/ It was :
So it comes down to what I stated before : one of DISTINCTLY FASTER and one just has pressure from Google (ie engineering decision or management decision) |
Wow this is sure spiraling - please keep it civil so the thread doesn't get locked. Since I wrote the referenced Android blog let me add some thoughts... When we look at the experience of editing a build and using a build there are many factors, performance being just one. For build editing, many developers I've talked to have ultimately faster iterations with KTS than with Groovy due to better help in the IDE (red squiggles if something is invalid, etc). The benchmarks here don't show the end-to-end developer feedback loop time. When using a build, KTS has shown to be sufficiently on-par with Groovy. KTS also gets some benefits from better build cacheability (ie remote build caches which I don't think are possible in the same way with Groovy). As been said before, users can choose Groovy or KTS. Android went KTS as a default for a number of reasons, including, Android is Kotlin-First from build to UI. There is a significant downside to developers needing to know multiple languages. And we've seen from many Android users that satisfaction is higher with KTS. However, if Android users want to use Groovy for their build, that is totally fine. |
Right... but Android isn't Gradle. Android is Google/Alphabet. I don't speak for all users only myself but I pretty sure not all users are Google/Alphabet; and I think Gradle is used for other things (not sure). Its the 'Microsoft Explorer by default for all' decision because Google 'decided' This is why I am asking if this was an engineering decision or a management decision? Because after seeing these benchmarks again and again, this CAN'T be an engineering decision. I mean I don't see any attempt at an engineering discussion between the two. |
Back in 2022, Gradle, JetBrains, and Android engineers all got together and decided together that we would all switch our defaults. Some reasons were different but ultimately we all decided together that this was the best default. |
The performance numbers represented here may be the only thing some users care about, but they aren't the only thing that every user cares about. |
I saw some Google engineers posts some of those reasons in their Google Gradle forum; some of it is FUD. Its concerning that misinformation can lead to decision that effect everyone And the performance numbers were enough to bother Google that they put warnings on their site. Thats seem like it bothered mre than a 'few'. |
An example, its one of the hottest topics to talk about in Kotlin... https://dev.to/cdsap/performance-impact-analysis-of-gradle-87-in-android-projects-5288 |
Gradle 8.7 enables remote build caching for Groovy build scripts: https://docs.gradle.org/8.7/release-notes.html#avoiding-groovy-build-script-compilation-thanks-to-build-cache-support |
Expected Behavior
The Kotlin and Groovy DSLs have as much as possible similar performance characteristics.
IOW there's no outlier use case for which the performance characteristics make using one or the other DSL a penalty.
Current Behavior
What follows is the current state of affairs as Gradle 6.8 for four use cases:
The data below comes from performance tests run on the Gradle CI. Those tests use a build with a large number of projects and build scripts, the build scripts being rather simple, and they run on both the Groovy and Kotlin DSL for comparison.
buildSrc
abi changebuildSrc
non-abi changeContext
The numbers above are, at the time of writing, for current
release
branch of Gradle that is the upcoming 6.8.1. See:Use Login as guest and note that these links will eventually get outdated.
This issue is a follow up to gradle/kotlin-dsl-samples#902 which resides in an archived repository.
One of the reasons the Kotlin DSL is slower when it is is the time it takes to the Kotlin compiler to compile the scripts.
See https://youtrack.jetbrains.com/issue/KT-24668
The text was updated successfully, but these errors were encountered: