初学rust,踩坑私有依赖
Tag rust, 依赖, git, on by view 164

最近尝试在一个项目中引用一个私有crate,这个crate是一个lib,然后存储在私有git仓库中,并且这个git平台不支持rust的crate-index.

  • 第一坑,引用git依赖

于是我在我的项目中这样引用外部依赖包

[package]
name = "las"
version = "0.1.1"
authors = ["...."]
edition = "2018"

[dependencies]
elasticsearch = { version = "8.5.0-alpha.1", default-features = false, features = [
    "rustls-tls",
] }
logwatcher = "0.1.1"
json = "*"
reqwest = { version = "0.11.12", default-features = false, features = [
    "blocking",
    "rustls-tls",
] }
toml = "0.8.6"
serde_derive = "1.0.177"
serde = "1.0.177"
serde_json = "1.0.104"
once_cell = "1.18.0"
futures = "0.3"
tokio = { version = "*", features = ["full"] }
uuid = { version = "1.4", features = ["v4", "fast-rng", "macro-diagnostics"] }
chrono = "0.4"
core_affinity = "0.8.0"
geoip = { git = "ssh://git@git.xxxx.com/xxxx/tgeoip.git", branch = "master", package = "geoip" }

nix = "0.26"
libc = "0.2.146"
clap = { version = "4.4.2", features = ["derive"] }
log4rs = { version = "1.2.0" }
log = { version = "0.4.20" }
[build-dependencies]
regex = "1.6.0"

需要注意的是,我git@git.xxxx.com:xxxx/tgeoip.git仓库中是一个workspace,里面有2个子项,一个是可执行bin项目,叫做regen,一个是外部可依赖包,叫做geoip(后续实践表明同一个workspace里面的子项引用就应该这么干,git路径是同一个,使用package指定子项的包名),我这样引入之后,发现死活拉取不了私有git依赖。报错没权限拉取,git鉴权失败

error: failed to get `geoip` as a dependency of package `las v0.1.1 (/data/code/rust/las)`

Caused by:
  failed to load source for dependency `geoip`

Caused by:
  Unable to update ssh://git@git.xxxx.com/xxxx/tgeoip.git?branch=master#e41c5279

Caused by:
  failed to clone into: /root/.cargo/git/db/tgeoip-9094aceea5940357

Caused by:
  failed to authenticate when downloading repository

  * attempted ssh-agent authentication, but no usernames succeeded: `git`

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  no authentication methods succeeded

各方查询后,都是建议添加这个选项,我开始还没搞清楚加在哪儿,但是发现使用环境变量好使。各方尝试后:

添加到~/.cargo/config文件中,如下

[net]
git-fetch-with-cli = true

然后再次执行cargo build无需环境变量即可。(前提是你本地已经配置好了访问ssh://git@git.xxxx.com/xxxx/tgeoip.git的ssh key)

  • 第二坑,外部依赖包要申明为rlib

我在这个bin项目中引用geoip,发现一直报错

failed to resolve: use of undeclared crate or module `geoip`

百思不得其解,最后发现是我依赖包申明类型的问题,我Cargo.toml申明的是

[package]
....

[lib]
name = "geoip"
crate-type = ["staticlib", "cdylib"]

[dependencies]
bincode = "1.3.3"
serde = { version = "1.0.193", features = ["derive"] }

实际lib应该申明rlib类型,才能被正常引用,改为如下,更新依赖,就没报错了

[package]
....

[lib]
name = "geoip"
crate-type = ["staticlib", "cdylib", "rlib"]

[dependencies]
bincode = "1.3.3"
serde = { version = "1.0.193", features = ["derive"] }

Bazel对依赖库进行patch遇到的问题
Tag bazel, patch, 依赖, on by view 149

最近使用bazel需要对一个go第三方依赖进行patch,这个依赖是gorm/v1,原作者对v1已经放弃了,我们项目中使用了v1,但是因为v1到v2是不兼容的升级,所以我们项目中无法升级。针对v1的优化和bug修复,就只能使用patch的方式进行了。

第一步将github.com/jinzhu/gorm克隆下来,check出最后一个版本v1.9.16,然后做修改。修改完毕之后,不要进行git提交,执行git diff >> new1.patch将会生成这次变更的patch文件。

将patch文件存储如下0001-go-errors.patch

third_party_patches
└── com_github_jinzhu_gorm
    ├── 0001-go-errors.patch
    ├── 0002-fix-sqlite.patch
    └── BUILD.bazel

其中,BUILD.bazel空文件即可。另外deps.bzl中的引用如下

go_repository(
    name = "com_github_jinzhu_gorm",
    build_file_proto_mode = "disable",
    importpath = "github.com/jinzhu/gorm",
    patch_args = ["-p1"],
    patches = [
        "//third_party_patches/com_github_jinzhu_gorm:0001-go-errors.patch",
        "//third_party_patches/com_github_jinzhu_gorm:0002-fix-sqlite.patch",
    ],
    sum = "h1:+IyIjPEABKRpsu/F8OvDPy9fyQlgsg2luMV2ZIH5i5o=",
    version = "v1.9.16",
)

但是我编译的时候,却报错了

➜  code git:(feature/migrate-flow) ✗ bazel build //...
INFO: Analyzed 74 targets (3 packages loaded, 141 targets configured).
INFO: Found 74 targets...
ERROR: /root/.cache/bazel/_bazel_root/aa05a35ce6b6cd2ce6b5a504fd9d9e22/external/com_github_jinzhu_gorm/BUILD.bazel:3:11: GoCompilePkg external/com_github_jinzhu_gorm/gorm.a failed: (Exit 1): builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -tags sqlite -src external/com_github_jinzhu_gorm/association.go -src ... (remaining 71 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
compilepkg: missing strict dependencies:
        /root/.cache/bazel/_bazel_root/aa05a35ce6b6cd2ce6b5a504fd9d9e22/sandbox/linux-sandbox/4877/execroot/ias-admin/external/com_github_jinzhu_gorm/main.go: import of "github.com/go-errors/errors"
No dependencies were provided.
Check that imports in Go sources match importpath attributes in deps.
INFO: Elapsed time: 0.843s, Critical Path: 0.06s
INFO: 4 processes: 4 internal.
FAILED: Build did NOT complete successfully

按照报错打开bazel cache路径,发现bazel为依赖生成的BUILD.bazel文件中,并没有添加我patch中引入的依赖。于是我将cache中的BUILD.bazel拷贝出来,手动加入依赖,然后git diff,得到BUILD.bazel的patch文件如下


diff --git a/BUILD.bazel b/BUILD.bazel
index 0849476..d26e9f7 100644
--- a/BUILD.bazel
+++ b/BUILD.bazel
@@ -32,7 +32,10 @@ go_library(
     ],
     importpath = "github.com/jinzhu/gorm",
     visibility = ["//visibility:public"],
-    deps = ["@com_github_jinzhu_inflection//:inflection"],
+    deps = [
+        "@com_github_jinzhu_inflection//:inflection",
+        "@com_github_go_errors_errors//:go_default_library",
+    ],
 )
 
 alias(
@@ -76,5 +79,6 @@ go_test(
         "//dialects/sqlite",
         "@com_github_erikstmartin_go_testdb//:go-testdb",
         "@com_github_jinzhu_now//:now",
+        "@com_github_go_errors_errors//:go_default_library",
     ],
 )

加入到我们的patch中,然后再次编译,就正常了。

我推断bazel patch这里面的逻辑是,先拉下原始包,第二步生成BUILD.bazel,最后再patch。google的研发bazel的人简直就是没长脑子,这样的步骤我还要它生成BUILD.bazel干啥,还不如直接把BUILD.bazel生成到patch文件里。总之,折腾了许久总算解决了,记录一下。