Option to omit source code all together

[alecthomas]

In very large repositories, I don't really want to include the source code again, I'd rather just link to the original source code files.

I realise this will require some level of heuristics for different source control hosters like GitHub, GitLab, etc.

[zigo101]

This needs some time to implement but I think it is possible. I will notify you when it is done (or the idea is given up).

Thanks for the suggestion.

[zigo101]

@alecthomas

Where is your source code hosted? Is it sufficient to only support github and gitlab?

[zigo101]

BTW, there is an obstacle that it is hard to detect the source hosts of third-party dependency packages. So the current implementation will only support external source code pages for standard packages and the current project packages.

[zigo101]

I just found a way to find out the source hosts of third-party dependency packages. But this way needs external network connection. Is it acceptable in your use case?

[alecthomas]

For sure, along with some kind of caching perhaps? Either way would be great though!

[zigo101]

Ah, maybe the host urls of the modules with github.com as import prefix could be possible to be guessed from their cache directory path. Though those modules with custom domain prefixes still need a https request to achieve the goal.

[zigo101]

@alecthomas
I just pushed a commit which supports popular code hosting websites and Go packages.
It is appreciated if you could run it on your code to check if there are still problems.

[alecthomas]

Will do this week hopefully, I'll let you know. Thank you very much.

[alecthomas]

Tried it out and it vastly reduced the size of the output, thank you!

It does still appear to generate source for the module itself though, even though it's on GitHub. Is that intentional?

[zigo101]

It is okay for my project, which is also hosted on Github.

I just pushed a new commit which will not panic for projects which are not git based. (unrelated to your last comment).

What are the outputs for the following 4 commands in your project directory?

git rev-parse --show-toplevel

git rev-parse HEAD

git rev-parse --abbrev-ref --symbolic-full-name @{upstream}

git remote get-url <the part before the slash of the output of the last command>

In my case, they are:

$ git rev-parse --show-toplevel
/home/myname/projects/go101/golds

$ git rev-parse HEAD
767f95dcac9fad16bc005d437d37b4b3fe823594

$ git rev-parse --abbrev-ref --symbolic-full-name @{upstream}
origin/develop

$ git remote get-url origin
https://github.com/go101/golds.git

[alecthomas]

My remote uses SSH because it's a private repo with an SSH based security model. It looks somewhat like this:

[email protected]:/<user>/<repo>.git

[zigo101]

OK, thanks for the information.

[zigo101]

I just pushed a new commit and tagged it as v0.2.7.
This commit should fix the problem now.

[zigo101]

BTW, the -allow-network-connection option could be used to detect the module repo URLs which are hard to guess.

[alecthomas]

Yeah I'm using that; works well.

[alecthomas]

(but does not work with the above)

[zigo101]

(but does not work with the above)

I don't very get it. Do you mean it doesn't work with the -allow-network-connection option?

[alecthomas]

With that option it reduces the output size from 900mb to 200m but still includes the source for my private ssh cloned repo.

[zigo101]

Some strange. What the output of with these options: golds -gen -v -source-code-reading=external -allow-network-connection -compact -s?

[zigo101]

@alecthomas
You don't neeed to paste all the output lines. There should be two related parts in the output:
One is the line contains (working directory) guess moudle , the other is the mobule info for
your module in the working directory, like

info_module.go:679: module: k8s.io/kubernetes@3da6c11ae5fc1ee7d6f992730fc7c1dfc81ddf81 (1127 pkgs)
info_module.go:680:             Pkgs[0].Dir: /home/USER/opensource/k8s.io/kubernetes/cluster/gce/gci
info_module.go:681:                     Dir: /home/USER/opensource/k8s.io/kubernetes
info_module.go:682:           RepositoryDir: /home/USER/opensource/k8s.io/kubernetes
info_module.go:683:           RepositoryURL: https://github.com/kubernetes/kubernetes
info_module.go:684:               ExtraPath:

[zigo101]

I think I get the reason, but I'm not very sure. Cloud you add the option -nouses and check whether the generated docs still includes your private source? This option should be turned off when external source links are enabled.

[zigo101]

Ah, the last comment is not valid.

I just pushed another commit, which fixed a bug which makes the source on "bitbucket.org" unable to be linked externally.

[zigo101]

With that option it reduces the output size from 900mb to 200m but still includes the source for my private ssh cloned repo.

Does the problem still exist with the tip? If the problem is gone, I will create the v0.2.8 tag.

[alecthomas]

I'll try to remember to check later today. No worries cutting a release though, the worst that happens is it uses a bit of extra data.