Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Miscellaneous smaller Pub serialization improvements #9287

Merged
merged 8 commits into from
Oct 21, 2024
Merged

Miscellaneous smaller Pub serialization improvements #9287

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
ff8bbc6
chore: Align custom kotlinx-serializers to be objects, part 2
sschuberth Oct 15, 2024
e59225e
docs(pub): Add links to dependency types
sschuberth Oct 15, 2024
6e834c3
chore(pub): Order dependency classes as in the linked documentation
sschuberth Oct 15, 2024
8be16d9
refactor(pub): Reduce code by delegating to the default serializer
sschuberth Oct 15, 2024
ae06cfc
chore(pub): Simplify deserializing dependencies
sschuberth Oct 15, 2024
4a3c230
chore(pub): Handle dependency types in the same order as documented
sschuberth Oct 15, 2024
e045fbd
refactor(pub): Reorder classes into packages
sschuberth Oct 17, 2024
c643da0
refactor(pub): Only use a single shared YAML instance
sschuberth Oct 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions plugins/package-managers/pub/src/main/kotlin/Pub.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,13 @@ import org.ossreviewtoolkit.model.config.AnalyzerConfiguration
import org.ossreviewtoolkit.model.config.PackageManagerConfiguration
import org.ossreviewtoolkit.model.config.RepositoryConfiguration
import org.ossreviewtoolkit.model.createAndLogIssue
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.Dependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.SdkDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.utils.PubCacheReader
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Lockfile
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.PackageInfo
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.Dependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.SdkDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.parseLockfile
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.parsePubspec
import org.ossreviewtoolkit.utils.common.CommandLineTool
import org.ossreviewtoolkit.utils.common.Os
import org.ossreviewtoolkit.utils.common.ProcessCapture
Expand Down
4 changes: 2 additions & 2 deletions ...b/src/main/kotlin/utils/PubCacheReader.kt → ...ers/pub/src/main/kotlin/PubCacheReader.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,14 @@
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.packagemanagers.pub.utils
package org.ossreviewtoolkit.plugins.packagemanagers.pub

import java.io.File

import org.apache.logging.log4j.kotlin.logger

import org.ossreviewtoolkit.downloader.VcsHost
import org.ossreviewtoolkit.plugins.packagemanagers.pub.PackageInfo
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.PackageInfo
import org.ossreviewtoolkit.utils.common.Os
import org.ossreviewtoolkit.utils.common.isSymbolicLink

Expand Down
10 changes: 3 additions & 7 deletions ...-managers/pub/src/main/kotlin/Lockfile.kt → ...ers/pub/src/main/kotlin/model/Lockfile.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,8 @@
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.packagemanagers.pub
package org.ossreviewtoolkit.plugins.packagemanagers.pub.model

import com.charleskorn.kaml.Yaml
import com.charleskorn.kaml.YamlConfiguration
import com.charleskorn.kaml.YamlInput
import com.charleskorn.kaml.YamlScalar

Expand All @@ -39,9 +37,7 @@ import kotlinx.serialization.descriptors.SerialKind
import kotlinx.serialization.descriptors.buildSerialDescriptor
import kotlinx.serialization.encoding.Decoder

import org.ossreviewtoolkit.plugins.packagemanagers.pub.PackageInfo.Description

private val YAML = Yaml(configuration = YamlConfiguration(strictMode = false))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, personally sacrificing encapsulation for these few bytes of memory is not worth it.
What do you think about the middle ground of keeping it private, but turning it into a function?

Copy link
Member Author

@sschuberth sschuberth Oct 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would encapsulation even be required here? I mean, these two files are supposed to be deserialized in exactly the same way, so it makes sense and is correct to share the same Yaml instance, or?

Background: I was originally doing this in the context of adding more tests for deserialization (which turned out to be unnecessary). As part of that, I needed access to YAML, but I couldn't make either instance internal as it would then have conflicted with the other private one.

import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.PackageInfo.Description

internal fun parseLockfile(lockfile: File) = YAML.decodeFromString<Lockfile>(lockfile.readText())

Expand Down Expand Up @@ -76,7 +72,7 @@ internal data class PackageInfo(
)
}

private class DescriptionDeserializer : KSerializer<Description> by Description.generatedSerializer() {
private object DescriptionDeserializer : KSerializer<Description> by Description.generatedSerializer() {
@OptIn(InternalSerializationApi::class)
override val descriptor: SerialDescriptor by lazy {
val serialName = checkNotNull(Description::class.qualifiedName)
Expand Down
67 changes: 28 additions & 39 deletions ...e-managers/pub/src/main/kotlin/Pubspec.kt → ...gers/pub/src/main/kotlin/model/Pubspec.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,8 @@
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.packagemanagers.pub
package org.ossreviewtoolkit.plugins.packagemanagers.pub.model

import com.charleskorn.kaml.Yaml
import com.charleskorn.kaml.YamlConfiguration
import com.charleskorn.kaml.YamlInput
import com.charleskorn.kaml.YamlMap
import com.charleskorn.kaml.YamlNode
Expand All @@ -35,17 +33,14 @@
import kotlinx.serialization.Serializable
import kotlinx.serialization.SerializationException
import kotlinx.serialization.decodeFromString
import kotlinx.serialization.descriptors.serialDescriptor
import kotlinx.serialization.encoding.Decoder
import kotlinx.serialization.encoding.Encoder
import kotlinx.serialization.serializer

import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.Dependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.GitDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.HostedDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.PathDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.SdkDependency

private val YAML = Yaml(configuration = YamlConfiguration(strictMode = false))
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.Dependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.GitDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.HostedDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.PathDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.SdkDependency

internal fun parsePubspec(pubspecFile: File): Pubspec = parsePubspec(pubspecFile.readText())

Expand Down Expand Up @@ -73,49 +68,43 @@
@Serializable
sealed interface Dependency

/** See https://dart.dev/tools/pub/dependencies#hosted-packages. */
@Serializable
data class HostedDependency(
val version: String,
val url: String? = null
) : Dependency

/** See https://dart.dev/tools/pub/dependencies#git-packages. */
@Serializable

Check warning on line 79 in plugins/package-managers/pub/src/main/kotlin/model/Pubspec.kt

View check run for this annotation

Codecov / codecov/patch

plugins/package-managers/pub/src/main/kotlin/model/Pubspec.kt#L79 

Added line #L79 was not covered by tests
data class GitDependency(
val url: String,
val path: String? = null,
val ref: String? = null
) : Dependency

/** See https://dart.dev/tools/pub/dependencies#path-packages. */
@Serializable
data class PathDependency(
val path: String
) : Dependency

/** See https://dart.dev/tools/pub/dependencies#sdk. */
@Serializable
data class SdkDependency(
val sdk: String
) : Dependency

@Serializable
data class GitDependency(
val url: String,
val path: String? = null,
val ref: String? = null
) : Dependency
}

/**
* If transformations like for JSON were available in kaml, this serializer could be simplified, see also
* https://github.com/charleskorn/kaml/issues/29.
*/
private object DependencyMapSerializer : KSerializer<Map<String, Dependency>> {
override val descriptor = serialDescriptor<Map<String, Dependency>>()

override fun serialize(encoder: Encoder, value: Map<String, Dependency>) {
TODO("Not implemented yet.")
}

private object DependencyMapSerializer : KSerializer<Map<String, Dependency>> by serializer<Map<String, Dependency>>() {
override fun deserialize(decoder: Decoder): Map<String, Dependency> {
val input = decoder.beginStructure(descriptor) as YamlInput

val result = when (val node = input.node) {
is YamlScalar -> emptyMap()
is YamlMap -> node.entries.asSequence().associateBy({ it.key.content }, { it.value.decodeDependency() })
else -> throw SerializationException("Unexpected YAML node type: ${node.javaClass.simpleName}.")
}
val result = input.node.yamlMap.entries.asSequence().associate { it.key.content to it.value.decodeDependency() }

input.endStructure(descriptor)

Expand All @@ -125,14 +114,6 @@
private fun YamlNode.decodeDependency(): Dependency {
if (this is YamlScalar) return HostedDependency(yamlScalar.content)

yamlMap.get<YamlScalar>("sdk")?.let { sdk ->
return SdkDependency(sdk = sdk.content)
}

yamlMap.get<YamlScalar>("path")?.let { path ->
return PathDependency(path = path.content)
}

yamlMap.get<YamlNode>("hosted")?.let { hosted ->
val version = checkNotNull(yamlMap.get<YamlScalar>("version")).content
val url = if (hosted is YamlMap) {
Expand All @@ -156,6 +137,14 @@
}
}

yamlMap.get<YamlScalar>("path")?.let { path ->
return PathDependency(path = path.content)
}

yamlMap.get<YamlScalar>("sdk")?.let { sdk ->
return SdkDependency(sdk = sdk.content)
}

throw SerializationException("Unexpected dependency node format.")
}
}
25 changes: 25 additions & 0 deletions plugins/package-managers/pub/src/main/kotlin/model/Yaml.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
/*
* Copyright (C) 2024 The ORT Project Authors (see <https://github.com/oss-review-toolkit/ort/blob/main/NOTICE>)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* SPDX-License-Identifier: Apache-2.0
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.packagemanagers.pub.model

import com.charleskorn.kaml.Yaml
import com.charleskorn.kaml.YamlConfiguration

internal val YAML = Yaml(configuration = YamlConfiguration(strictMode = false))
4 changes: 2 additions & 2 deletions ...c/test/kotlin/utils/PubCacheReaderTest.kt → ...pub/src/test/kotlin/PubCacheReaderTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,15 @@
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.packagemanagers.pub.utils
package org.ossreviewtoolkit.plugins.packagemanagers.pub

import io.kotest.core.spec.style.WordSpec
import io.kotest.engine.spec.tempdir
import io.kotest.matchers.shouldBe

import java.io.File

import org.ossreviewtoolkit.plugins.packagemanagers.pub.PackageInfo
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.PackageInfo
import org.ossreviewtoolkit.utils.common.Os
import org.ossreviewtoolkit.utils.common.safeMkdirs

Expand All @@ -35,7 +35,7 @@
private const val RELATIVE_PATH = ".."
private val ABSOLUTE_PATH = File(Os.env["PUB_CACHE"])

class PubCacheReaderTest : WordSpec({

Check warning on line 38 in plugins/package-managers/pub/src/test/kotlin/PubCacheReaderTest.kt

View workflow job for this annotation

GitHub Actions / qodana-scan

Unused symbol 

Class "PubCacheReaderTest" is never used
val tmpPubCacheDir = tempdir().also { Os.env["PUB_CACHE"] = it.absolutePath }
val gitPackageCacheDir = tmpPubCacheDir.resolve("git/$PACKAGE_NAME-$RESOLVED_REF")
val gitPackageWithPathCacheDir = tmpPubCacheDir.resolve("git/$PACKAGE_NAME-$RESOLVED_REF/$PACKAGE_NAME")
Expand Down
10 changes: 5 additions & 5 deletions ...nagers/pub/src/test/kotlin/PubspecTest.kt → .../pub/src/test/kotlin/model/PubspecTest.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,17 @@
* License-Filename: LICENSE
*/

package org.ossreviewtoolkit.plugins.packagemanagers.pub
package org.ossreviewtoolkit.plugins.packagemanagers.pub.model

import io.kotest.core.spec.style.WordSpec
import io.kotest.matchers.nulls.beNull
import io.kotest.matchers.should
import io.kotest.matchers.shouldBe

import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.GitDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.HostedDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.PathDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.Pubspec.SdkDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.GitDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.HostedDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.PathDependency
import org.ossreviewtoolkit.plugins.packagemanagers.pub.model.Pubspec.SdkDependency

class PubspecTest : WordSpec({
"parsePubspec()" should {
Expand Down
Loading