Mine Rust crates

[pombredanne]

We need to collect all the crates and index them.
There is an API on crates.io and also a git repo with an index:

• https://github.com/rust-lang/crates.io-index

[alok1304]

@pombredanne, suppose we collected and indexing all the crates from Crates.io. Once we have all crates indexed, could you clarify the next steps? Should we focus on setting up querying functionality, integrating the index into PurlDB, or enriching the data in any particular way? Any guidance on priorities or additional requirements would be greatly appreciated!

[pombredanne]

@alok1304 re:

suppose we collected and indexing all the crates from Crates.io. Once we have all crates indexed, could you clarify the next steps? Should we focus on setting up querying functionality, integrating the index into PurlDB, or enriching the data in any particular way? Any guidance on priorities or additional requirements would be greatly appreciated!

Getting the index is straightforward, this is just a git repo. With this the steps would be:

For batch operations:

1. Parse the index and populate the PurlDB with basic cargo packages
2. For one package from 1., further collect detailed metadata from the API
3. For one package from 2., further integrate the download and scan of the crate code

For the on-demand API endpoint:

1. For one package PURL as an input, do the 2. and 3. from above, on demand

This would establish the data in PurlDB to collect, index, match and have a comprehensive references for all the crates.

[pombredanne]

In terms of priority, the on-demand operations are likely higher priority to get some results fast to than the full batch operations that come afterwards. Unless Batch 1. is really easy, which it may be.