node-list software

[makew0rld]

^{This initial comment is collaborative and open to modification by all.}

Task Summary

🎟️ Re-ticketed from: #57
📅 Due date: N/A
🎯 Success criteria: Write software that converts node-list JSON into desired outputs, and set it up in the node-list CI.

Hmm, would be cool if there is a library for:

• coordinates -> kml
• coordinates -> map PNG

Then we can travis ci run them and output assets to github releases on merge.

From @benhylau here

To Do

• KML
  • One option would be Python using simplekml (docs, Github)
• GeoJSON
  • Likely just simple Python JSON manipulation can achieve this
• Map images
  • If we use OSM and Leaflet there are some options
  • Mapnik, possibly with Nik4
    • Tool for rendering maps
    • Can render KML on top of OSM maps
      • https://stackoverflow.com/a/3979536/7361270
      • https://stackoverflow.com/questions/15691525/python-mapnik-example-on-how-to-render-a-map-with-a-gps-track-on-it
    • Likely install mapnik2
  • PAT is another option, doesn't seem to be well documented though
• Setup CI for all the above outputs
  • Decide where these outputs will be stored:
    • In a branch without history, gh-pages style. No history of outputs, but a simpler model
    • Have a Github release for every commit, and upload them as release assets
    • Both, actually

Other outputs? Feel free to edit and add.

[darkdrgn2k]

@Shrinks99 suggetsd GeoJSON or similar as well

I wrote a quick javascript script to do the conversion to illustrate how it could be done in real time
https://cryptpad.fr/code/#/2/code/view/eOFDZxhooqTh8bn2jdOm+1p8-wpNfOuNUquQiMj4-sU/

I don't know what the real-world application of it was but i know he was interested in int. perhaps he can comment

[makew0rld]

I don't really have a sense of who GeoJSON would be useful for (any ideas?), but it seems easy enough to add as an output. Thanks for the code!

[darkdrgn2k]

don't know maybe @Shrinks99 can advise

[Shrinks99]

GeoJSON is a somewhat popular spec for mapping applications. If we want to convert our map to use Open Street Map eventually we'll likely need to have this format.

That said, this is not really a super high priority at the moment however I would appreciate it for the web I've been working on here and there. Nothing concrete or equivalent to what we have right now though.

Regarding image creation, Leaflet (the OSM interface I've been using) does have some options for that. Check them out! Generally good images are easier to get out of OSM than Google,

[makew0rld]

Interesting, thanks. GeoJSON doesn't seem that hard to convert to, and could even become the new internal format. For now it can just be an output though.

Those leaflet options seem interesting too, potentially exactly what we want. Although probably not useful with the current system because we don't use OSM. I think switching to OSM might be out of the scope of this issue, but if there's no good way to get Google Maps images we might be forced to make the switch.

[Shrinks99]

Already brought up the idea of just making it the default format, @darkdrgn2k explained that trying to fit the data to your use case rather than the other way around was a bad way to go and I agree.

Switching to OSM is out of scope of this issue, I can show you what I've been working on and using it for printing images might be a good first step at eventually ditching Google. We've got more important things to do though.

[makew0rld]

Waiting on new schema to be defined in #57.

Edit: Code will be based on schema v0.7, being defined in this PR.

[benhylau]

The only other I can think of is yml but not sure if huge value there. Your current list seems solid.

[makew0rld]

KML script is finished, see here. I will write the GeoJSON output (and maybe CI stuff) before creating a PR.

Edit: GeoJSON too..

[makew0rld]

PR created here.

As of 60b7949 KML and GeoJSON output is working.

[makew0rld]

Current issue: The release URLs cannot be hotlinked for use in other applications due to GitHub's Content Security Policy. Web browsers will not make the request.

I think we should combine both proposed models and also keep the latest versions of the outputs in a branch with no history, gh-pages style. This will allow hotlinking.

Edit: Complete, see the README for direct links, and the assets branch.

[benhylau]

First of all, congrats on getting the pipe set up!

Can render KML on top of OSM maps: stackoverflow

This may auto-generate assets for us to distribute in a region around the nodes!

Also, I propose to name files more descriptively and consistently, for example:

• tomeshnet-node-list-geojson.json
• tomeshnet-node-list-kml.kml

Even the source file nodeList.json should use tomeshnet-node-list.json instead.

[makew0rld]

This may auto-generate assets for us to distribute in a region around the nodes!

That's the idea! 🤞 It looks somewhat difficult, but I will try and get that set up.

Also, I propose to name files more descriptively and consistently, for example:

• tomeshnet-node-list-geojson.json
• tomeshnet-node-list-kml.kml

Done, see this commit.

Even the source file nodeList.json should use tomeshnet-node-list.json instead.

I'm not going to change this now since other things rely on it, but that would be nice.

[benhylau]

@YurkoWasHere outside of this repo, is this the only dependency?

@makeworld-the-better-one if so I'd say now is the best time to change it :)

[makew0rld]

I believe so. I'll wait on confirmation before changing so that the website can be updated at the same time.

[darkdrgn2k]

There are several other places the use nodeList.json but i dont think they are officially.

• DNS script
• CJDNS Walker (now defunct)
• Prometheus pull (probably now defunct)

i should sunset/cleanup the rest anyway since the format has changed drastically. but i would be fine in changing the name.

[darkdrgn2k]

I would like some clarity on the point of tomeshnet-node-list.json this would inform the naming convention too.

• Will this include end-user nodes (CPE etc)
  • will it only be community managed nodes
  • If so where will end users nodes live
  • If not are we going to include proposed nodes in here
• Are we going to include services into the mix

[makew0rld]

I have opened tomeshnet/node-list#44 with the name change.

• Will this include end-user nodes (CPE etc)
• Are we going to include proposed nodes in here

I would say yes for now, to both things. We can expand it if needed, but I don't see why not.

Are we going to include services into the mix

I've mentioned this in tomeshnet/node-list#25 before, I think that having a separate services file is a good idea, as it tends to be used for different things than the list of nodes.

[benhylau]

I would say, put every node type into the list and make the rest a future problem. Services not at the moment.