Jbrowse2

[fubar2]

FOR CONTRIBUTOR:

• - I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
• - License permits unrestricted use (educational + commercial)
• - This PR adds a new tool or tool collection

This is a very stripped-down working prototype, repurposing @hexylena's JBrowse1-in-Galaxy for JBrowse2.

Available for testing from the toolshed

Most things are working in testing - screenshot shows bed bam vcf maf bigwig gff sample data tracks

and HiC

• JB1 is EOL.
• JB2 configuration json is wildly different from JB1.
• JB2 defaults are sensible enough for many non-complicated situations.
  • for many workflows like the TreeValGal project, this will work very well as is
  • some challenges using jbrowse1 with very dense tracks in huge VGP data.
• Only the easy part is working for the basic tracks, recycling Helena's core code, and adding hic.
• URLs are used wherever easy to do to minimise the size of JBrowse2 files folder so these are much smaller than jb1 where all were copied. Not practicable for huge files like hic. JBrowse2 lets galaxy offload existing data and sequence files to nginx range downloads.
• A lot of work remains, to figure out and implement all the new track configuration options in the UI.
  • Could not find any human readable documentation on these - need to use the browser to find track setting minutae.
  • Nor was there anything on changing to different codon schemes that I could find. So that's a big loss - but it might be buried somewhere. The new architecture is plugins to do stuff so might need one for codons.
  • SPARQL, REST remain to be implemented.

[bgruening]

Can you please revert this change to the older jbrowse?

[fubar2]

yes - I did as soon as I saw planemo testing the wrong xml

+ GITHUB_REF=refs/pull/5695/merge
+ '[' 'tools/jbrowse
tools/jbrowse2' == '' ']'
+ echo -n 'tools/jbrowse
tools/jbrowse2'
+ echo -n tools/jbrowse2/jbrowse2.xml
+ echo -n 20
+ '[' lint == lint ']'
+ lint_fail=false
+ mapfile -t REPO_ARRAY
+ for DIR in "${REPO_ARRAY[@]}"
+ '[' false '!=' true ']'
+ planemo shed_lint --tools --ensure_metadata --urls --report_level all --fail_level warn --recursive tools/jbrowse
+ tee -a lint_report.txt
Linting repository /home/runner/work/tools-iuc/tools-iuc/tools/jbrowse

[fubar2]

oh I rewrote the wrong one too ! Sorry - yes of course.

[fubar2]

done @bgruening Sorry...

[fubar2]

Still seeing CI lint failing with the wrong xml - planemo is linting jbrowse not jbrowse2 @bgruening. Not sure what to do.

tools/jbrowse2' == '' ']'
+ echo -n 'tools/jbrowse
tools/jbrowse2'
+ echo -n tools/jbrowse2/jbrowse2.xml
+ echo -n 20
+ '[' lint == lint ']'
+ lint_fail=false
+ mapfile -t REPO_ARRAY
+ for DIR in "${REPO_ARRAY[@]}"
+ '[' false '!=' true ']'
+ planemo shed_lint --tools --ensure_metadata --urls --report_level all --fail_level warn --recursive tools/jbrowse
+ tee -a lint_report.txt
Linting repository /home/runner/work/tools-iuc/tools-iuc/tools/jbrowse
Applying linter expansion... CHECK
.. INFO: Included files all found.

[fubar2]

ok...passes lint now. Ouch. That was annoying clumsiness.

[fubar2]

Odd. Passes planemo tests here:

d0c-e303-426f-8b32-8b8dd5b8df42.dat to /tmp/tmp9pxo1ggt/files/2/b/e/dataset_2bea3d0c-e303-426f-8b32-8b8dd5b8df42.dat
galaxy.model.metadata DEBUG 2023-12-27 21:00:47,645 [pN:main.1,p:3223405,tN:LocalRunner.work_thread-1] loading metadata from file for: HistoryDatasetAssociation 10
galaxy.jobs DEBUG 2023-12-27 21:00:47,719 [pN:main.1,p:3223405,tN:LocalRunner.work_thread-1] job_wrapper.finish for job 10 executed (380.086 ms)

Shut down
supervisord has terminated
All 4 test(s) executed passed.
jbrowse2 (Test #1): passed
jbrowse2 (Test #2): passed
jbrowse2 (Test #3): passed
jbrowse2 (Test #4): passed
(.venv) ross@pn50:~/rossgit/tools-iuc/tools/jbrowse2$ planemo --version
planemo, version 0.75.19

[bgruening]

Please note that the CI is running 'planemo test --biocontainers '

[abretaud]

Nice to see progress! Have you been able to reuse some code from #3997 or you restarted from scratch?

[bgruening]

@fubar2 tests are passing now again. Thanks to @mvdbeek!

[fubar2]

Nice to see progress! Have you been able to reuse some code from #3997 or you restarted from scratch?

@abretaud: Helena's JB1 code is extensively reused but unfortunately I didn't know you'd done all that before I started hacking on it...big difference from #3997 is there is an option to generate json directly rather than calling jbrowse to do it - both seem to work - the two options are there because it took me a while to realise that no matter how config.json is being generated, even if it works fine in Galaxy, when downloaded cannot be understoood by the desktop client. It does work when served by nginx of course - there seems to be a complete disconnect unfortunately - but all the common formats work and with a PR for bcbio, maf will work too.

[fubar2]

@bgruening: after a lot of pfaffing about, it turns out that the JBrowse2 desktop client cannot understand a setup that works perfectly well in Galaxy or when served locally by nginx.
It seems that as part of the upgrading process there is now a complete disconnect between the config.json required for the desktop and a web server.
Very annoying - we need to talk to them urgently... #4161 suggests this is on the radar but not likely to happen any time soon.

At present, JB2 works fine in Galaxy. Downloaded tarballs can be unpacked and served statically from nginx easily enough but cannot be used by the desktop client.
Advice appreciated :)

[fubar2]

@bgruening: the python code for a simple web server that understands byte ranges, serves an unpacked archive from this tool nicely.
Added to every output archive, so after unarchiving and changing to the root, the user runs:

python servjb2.py
Defaults are all to serve that local directory but it has optional parameters: --root [path to the unpacked archive. default cwd] --port [default 8080] --bind [default 0.0.0.0]

Best I can think of - it may be some time #4161 before there is harmonisation between jbrowse2 desktop and web.

[fubar2]

@bgruening: Stumped again. Passes without biocontainers but fails using biocontainers.

Command '['jbrowse', 'create', '-f', '/tmp/tmpa3pi9_9n/job_working_directory/000/2/outputs/dataset_755c5b88-c40d-429c-a1bf-fad6dbe1b72c_files']' returned non-zero exit status 1.

So it's not allowed to write to the jbrowse2 output folder - do I need to fix anything - this may be an os.realpath thing. Let me see.

[fubar2]

Tried copying the way it's done in jb1 - more revealing error message:

ERROR:jbrowse:b"[E::bgzf_index_dump] Error opening /tmp/tmpxo0borav/files/4/a/8/dataset_4a806178-9e13-4656-8ce6-a6a284d52795.dat.gz.gzi : Read-only file system\nCould not write index to '/tmp/tmpxo0borav/files/4/a/8/dataset_4a806178-9e13-4656-8ce6-a6a284d52795.dat.gz.gzi'\n"

[fubar2]

Closed - #5697 merges @abretaud's PR content.