[plugin] org-roam.nvim progress & discussion #702
Comments
|
File is considered valid only if it has an local bufnr = vim.api.nvim_create_buf(false, true)
vim.api.nvim_buf_set_option(bufnr, 'filetype', 'org')
vim.api.nvim_buf_set_name(bufnr, 'somename.org')
vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, { '* Heading 1-a', '** Heading 2', '* Heading 1-b' })
vim.cmd('b'..bufnr) |
|
@kristijanhusak that solved the error about not being a current file! I'm hitting an issue where the plugin still doesn't detect a fold. There may be something wrong with my buffer or setup. Applies on any heading.
|
|
What would you expect to be folded here? Did you try doing |
|
The first heading. If I create a file manually and reproduce the contents, I can fold it.
I had to make the repo private while I got permission to work on it on personal time. Now that I have it, here's the current plugin: https://github.com/chipsenkbeil/org-roam.nvim About the pluginDatabaseI wrote a simplistic implementation of a database with indexing supported that provides an easy search for links and backlinks. It isn't as full-fledged as SQL and I'm sure will struggle with larger roam collections, but for me this is enough. ParserLeveraging the orgmode treesitter parser for me to find the details I need to build the above database. Completion of node under cursorCovers both completing within a link and under a cursor. Essentially does the complete everywhere out of the box. Here's a demo: example-org-roam-completion.mp4Quickfix jump to backlinksI like the quickfix list, so while Emacs doesn't have this and uses an org-roam buffer, this was easy to whip up for neovim: example-org-roam-quickfix.mp4Org bufferI was generating an org file instead of writing a custom buffer. I may switch to the custom buffer highlighting and format anyway because I need to capture link navigation to open in a different buffer versus the current one. Id navigationDoesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently. So I'll need to write an org-roam variant to support opening id links. Shouldn't be hard at all. I also built out id generation for a uuid-v4 format in pure Lua. My test nodes aren't using it, though.
|
|
Thanks for opening it up. I just skimmed through it, and it seems you did a lot of custom stuff.
In the issue description you wrote you are using commit 651078a. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization. |
|
Thanks for the pointers! When I looked at the plugin's API and data structures, it was (and I believe still is) missing crucial information I'd need for a fully-functional org-roam implementation. Would it make sense to move this discussion to a separate issue? I know you have the plugin API pinned issue, but there's a lot of different questions and dialog I'd want to have about needs and usage that feels better for a separate issue.
Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under cursor, etc. Wonder how much of that has changed or was misinformed from my first look. I also didn't check to see how you're loading files. I tried to keep most of my logic async - in the sense of leveraging neovim's scheduler and callbacks - so I can (in the future) easily report loading a large set of files, monitoring file changes and reloading, etc.
I don't know on this one. The database I wrote is a way to both track outgoing links (ones you'd find in a file) and incoming links (aka backlinks into a file). I like the design I've got, so I'll most likely keep this. Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.
Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.
I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.
Good to know. I've got different versions on different machines. Given I started fiddling with this a couple of months back, I guess the org id was implemented after.
I want to reduce the hacks and redundancy where possible. My implementation is meant to build on top of your plugin to supply the org-roam features, but when I started it looked like there was enough missing that I ended up only leveraging the treesitter language tree to get the information I needed. Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam. |
|
We can either have a separate issue or rename this one.
You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files
This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.
I think you will be able to use it. You just need to create custom
Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through
I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it. |
chipsenkbeil
changed the title
Renamed this one.
I'll give these a look in a week or two to see how they operate.
I don't know if it makes sense to add it to that method, but the locations of links within a file are needed for org-roam in order to support listing multiple references to the same backlink and to be able to pull in a sample of the content that linked to a node. I think it's easier to see from this person's tutorial of the Emacs usage: https://youtu.be/AyhPmypHDEw?si=mLGsAdosnKjTZ-1f&t=1690
I was really hoping that I could reuse it, so this makes me happy to hear. :) Will be giving that a look as soon as I reach that point.
Got it. Yes, I definitely want to use your code to handle omnicompletion. The selector is more like a built-in telescope interface, which I wanted to have similar to what is shown in the emacs tutorial above when it comes to selecting nodes where you can not only select between choices but also filter the choices further. So having both is my plan.
I don't really mind where my plugin lives, so if you would be interested in taking this in at some point in the future once we remove as much of the custom logic as makes sense, then I'd be happy to hand it over to you. I need something like this for work, and it didn't exist, which is why I'm building it now. |
|
I added We can add a few more things as we go if you find them missing, just let me know. |
|
@kristijanhusak are these indexed starting at 0 or 1? |
|
It's 1. |
chipsenkbeil
changed the title
|
@kristijanhusak I just updated to c4eeb3d and when trying to open an id link using
Given this fact and your desire to maintain only core orgmode functionality within this plugin, I'm assuming that unless the global ID at a file level is part of core orgmode, I will need to write my own open logic for links that navigates to the correct id that supports file-level ids. The thought process is to have that function fall back to your implementation if the link is not an id link. Thoughts? |
|
I changed the org-roam buffer to more closely match the org-roam variation. Looks like they changed to use magit underneath, which I'm just replicating in style right now. Also, progress on following node change under cursor. Don't know the performance considerations of this given I've got tiny tests. cursor-following-buffer.mp4 |
|
Id links work fine for me. There's also a test that confirms that it's working. Note that your files need to be part of |
|
@kristijanhusak your test has a property drawer with an id that is within a section with a headline. Does it work with a top-level property drawer? 
The case I'm referring to is a top-level property drawer not within a section. 
My dotfiles configure every org file (including those of the roam directory) to be within the agenda: 
|
|
Ah yes, you are correct, there was no support for top-level id. I added that now on the latest master, let me know if it's not working as expected. |
|
@kristijanhusak fantastic! Quick test has it working just fine. :) One less thing I have to manage myself. |
|
@kristijanhusak I haven't been able to find it yet. Do you have a version of org-id-get-create (part of This is referenced in org-roam's manual and is used to inject an ID into a head as seen in this demo. |
|
Yes, you can use this https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua local id = require('orgmode.org.id').new()There's also a method on headline classid_get_or_create that adds id property to a headline if there isn't one already. |
This is what I'm specifically looking for, which both does the work of generating the id and placing it in the headline. Is there a version that also works with a property drawer at the file level? When I tested Here's an example: org-id-get-create.mp4 |
|
We could add that, but I don't think you will need it. For org-roam you will have custom templates that will already populate this information before showing the capture window. When creating a template, just call the function OrgRoamTemplate:compile()
--call parent compile
local content = OrgTemplate:compile()
content = vim.list_extend({
':PROPERTIES:',
':ID: '..require('orgmode.org.id').new(),
':END:'
}, content)
end |
|
@kristijanhusak I'm switching back to having the org-roam buffer be an actual org file w/ syntax. I've seen examples of org-roam that use magit as the interface and others where it has a plain org file. From the discussions I've seen regarding how the buffer is used, people will either use it to open a backlink/citation/unlinked reference directly in the frame (via I know OrgMappings:open_at_point() exists to open the link or date at the point given. Is there any function that lets you open at point while specifying a different window? Was looking at that function and considering if I need to build a wrapper for that function to be able to point to a different window. Org-roam manual referenceLink: https://www.orgroam.com/manual.html#Configuring-the-Org_002droam-buffer-display
Emacs manual referenceLink: https://orgmode.org/org.html#Handling-Links There's a minor reference to
There's a minor reference to
Other references |
|
We could extend it to accept the command that opens the "point", and default it to |
I think that should work, yeah. As long as there is a way for me to specify the window to use versus the current window. I think I'd do this by using ---@param winnr integer # id of window where we will open buffer
---@return string # to supply to vim.cmd(...)
function make_cmd(winnr)
local cmd = winnr .. "wincmd w" -- goes to window "winnr"
return cmd .. " | edit"
endUpdateLeveraging loading a singular orgfile per preview in the org buffer. I was reading through how org-roam does this (via changelog and source), and it looks like it handles certain cases specially such as detecting if the link is in a heading and just showing the heading text or detecting if the link is in a list item and showing the entire list. I'll be switching this back over to an org buffer soon, but here's a preview:
This is my logic for now, although I'm assuming I should create a new -- NOTE: Loading a file cannot be done within the results of a stat,
-- so we need to schedule followup work.
vim.schedule(function()
require("orgmode.files")
:new({ paths = opts.path })
:load_file(opts.path)
:next(function(file)
---@cast file OrgFile
-- Figure out where we are located as there are several situations
-- where we load content differently to preview:
--
-- 1. If we are in a list, we return the entire list (list)
-- 2. If we are in a heading, we return the heading's text (item)
-- 3. If we are in a paragraph, we return the entire paragraph (paragraph)
-- 4. If we are in a drawer, we return the entire drawer (drawer)
-- 5. If we are in a property drawer, we return the entire drawer (property_drawer)
-- 5. If we are in a table, we return the entire table (table)
-- 5. Otherwise, just return the line where we are
local node = file:get_node_at_cursor({ opts.row, opts.col - 1 })
local container_types = {
"paragraph", "list", "item", "table", "drawer", "property_drawer",
}
while node and not vim.tbl_contains(container_types, node:type()) do
node = node:parent()
-- Check if we're actually in a list item and advance up out of paragraph
if node:type() == "paragraph" and node:parent():type() == "listitem" then
node = node:parent()
end
end
-- Load the text and split it by line
local text = file:get_node_text(node)
item.lines = vim.split(text, "\n", { plain = true })
item.queued = false
-- Schedule a re-render at this point
opts.emitter:emit(EVENTS.CACHE_UPDATED)
return file
end)
end) |
|
we could also make the argument a function, that would default to: local edit_cmd = edit_cmd or function(file) return 'edit '..file end
vim.cmd(edit_cmd(filename))That might give you more control over it.
Yes, you should go with the single instance. Here I have an Org instance that holds other instances. orgmode/lua/orgmode/files/file.lua Lines 108 to 119 in b7c42e6 |
I think either solution should work. Org roam bufferI'm still hitting some quirks with creating an org buffer that is of
example-of-org-buffer-issues.mp4 |
|
I would need to investigate how scratch buffer behaves. Could you try using a temp filename and see if it's better ( |
|
@kristijanhusak also, even with the changes you've made and the PR I submitted, I'm still having updates of the property fail. It looks like the end line/col may be off. If this is zero-indexed, line 6 is correct for the position of the property. So line 5 is incorrect. The buffer is the right one, though. 
I'm assuming this is because you're using I think you could replace Example Fix for
|
|
@kristijanhusak another issue I noticed that is unrelated. It seems like something my plugin is doing is preventing agenda from working. With my plugin installed, accessing the org agenda via I'm not within a roam file, but even if I was I wouldn't expect roam to interfere with agenda information. I do know that when I do something like Is there something I can debug to figure out why this is happening? Any theories regarding what I may have touched or be modifying that causes this issue?m Found the problemIt looks like the
Found the underlying problem & solutionTurns out,
You may want to scan through other cases of construction to see if it's accidentally using ---@class SomeClass
---@field some_field string
---@field other_field integer
local M = {}
M.__index = M
---Creates a new instance of SomeClass.
---@param opts {some_field:string}
---@return SomeClass
function M:new(opts)
local instance = {}
setmetatable(instance, M)
-- Populate fields for the instance
instance.some_field = opts.some_field
instance.other_field = 1234
instance:__do_extra_setup()
return instance
end |
|
I pushed fixes for the buf issues you opened PR for, and the fix for the file paths: 5875037 |
|
@kristijanhusak thanks for the speedy resolution on that one 😄 Another quirk I ran into by accident was when I tried to load an org file, but forgot to place the extension I ran into this issue because I thought the promise would only resolve on success, but it also resolved when returning |
|
Added types in ba2ab78 |
Great! I'm back from vacation, and refactoring the plugin to be more easily testable and been writing test coverage before the release. Nearly there. 😄 |
|
@kristijanhusak one quirk I've noticed for awhile, and am not sure if it's expected, is that directives don't parse if there is text immediately after them on the following line. In the recording below, I show the title that is calculated for a node. If there is a example-of-directive-not-working.mp4 |
|
It happens because once you add some content that is not a directive, it converts it to For example, if you have this: Is |
Yeah, that's what I assumed was happening, but the confusing aspect is that the directive is still highlighted like a directive instead of like part of the paragraph, which is why it was surprising that it didn't work earlier. Why is that the case if the treesitter parser doesn't treat it like a directive anymore?
My expectation would be that the directives work even with content above/below them because Emacs has plenty of examples of this including directives that expect something on the following line. And things like #+INCLUDE would definitely support being placed anywhere in the file.
And if I search github, while I'll find plenty of examples that include a newline between a series of directives and the start of content, I'll also find examples where directives have content immediately following them like this one. While I haven't tested anything specific in Emacs, the highlighting at least alludes to it being a directive.
|
Highlighting targets Generally, the parser has some issues figuring out when something is a paragraph and when something is a directive on the body level. I tried figuring out what can be the issue but I don't have that much experience writing parsers. If you have any idea please take a look at https://github.com/nvim-orgmode/tree-sitter-org. |
I can take a look, although I think the general solution for me is to query for directives anywhere similar to syntax highlighting, right? Is there a downside to doing that? They can show up anywhere in a document as long as they start the line. |
|
You can do that, but that might be misleading since directives can be used for a few other things like tables for example. What do you need a directive for? |
At the moment, simple things: title and file tags. From what I can see of the org roam manual, nothing else in v2 (the version I'm mirroring and extending) uses directives. I'm more thinking on the easiest way to find 
|
|
@kristijanhusak I ran into a weird bug. Haven't been able to reproduce it yet. I was in the directory interface of lir.nvim, deleted all of the files within, and then tried to open a capture buffer via the org roam api. The below error popped up, and after that I was presented with the template selection dialog like usual. 
|
|
@kristijanhusak 
Highlighting calendar datesWorking on the dailies extension of org roam, which should be a quick addition. Only thing I'm not sure about is if there's a way to specify in the calendar interface which days to highlight a certain color. With org roam, it does some injection to highlight days in a month that have a pre-existing note with a blue color versus the default black text. Any thoughts on how to do that with your calendar api? |
|
@kristijanhusak I know we've made changes to the capture API earlier. Is there any chance that one additional callback/handler could be supported that reports when a refile is canceled? I'm going through the org roam plugin and rewriting functions that use callbacks to instead return promises. The issue is that there's no way for me to fully resolve a promise that I'm creating tied to a capture, where I return the newly-created node's ID or Today, I'm using the pre_refile and post_refile callbacks for material we discussed. If there was a way to supply an on_cancel callback, then I'd be able to have a fully-resolvable promise wrapper around capturing an org roam node. |
|
Added |
@kristijanhusak thoughts on this one? |
|
@chipsenkbeil are you opening the calendar yourself or using some of the existing orgmode mappings? |
I'm invoking |
|
This is how today's date is highlighted We can add a hook like |
How would |
|
@kristijanhusak heads up, I assumed that I submitted a PR to update |
A thought I had for how to do this - but I want to get your perspective first - is to have a loop through all lines that uses a similar matching to your today check. The user of the calendar would register something like Just after the check for highlighting the current day, I could see this code within render: for i, line in ipairs(content) do
local from = 0
local to, num
-- Process all days on the line
while true do
from, to, num = line:find("%s(%d%d?)%s", from + 1)
if from == nil then break end
if from and to then
local date = Date:new({ year = self.month.year, month = self.month.month, day = num })
local hl = self.on_render_day(date)
-- If the user-provided function returns a highlight group, we apply it
-- Alternatively, the on_render_day could take the buffer, namespace, line, col_start, col_end to do its own thing
if hl then
vim.api.nvim_buf_add_highlight(self.buf, namespace, hl, i - 1, from - 1, to)
end
end
end
endThis could potentially replace the current logic to highlight today, and just have that be the default render day function. |
|
I had a slightly different idea, but this one also works. I'll refactor it later if necessary. return Calendar.new({
date = Date.now(),
on_day = function(day, opts)
if day:is_weekend() then
vim.api.nvim_buf_add_highlight(
opts.buf,
opts.namespace,
'WeekendHLGroup',
opts.line - 1,
opts.from - 1,
opts.to
)
end
end,
})
|
|
@kristijanhusak works like a charm! 
|
|
@kristijanhusak I can't think of anything else I need for the core roam functionality other than #718, which I know is a larger ask than you have time for at the moment. 😄 So with that being said, thoughts on closing this issue and #66? If possible, I'd love to have a newly-tagged version of the orgmode plugin, so I can update my README instructions to reference needing orgmode 0.3.x+ or 0.4+ instead of a specific commit. I've got a video in the works to run through the plugin, so will be excited to share that in the future! Thanks again for all of the support you've given me over the past two months! |
|
Created release https://github.com/nvim-orgmode/orgmode/releases/tag/0.3.4. |
About this task
This issue started discussing a potential bug, and ended up evolving into a rundown of the ongoing work on org-roam.nvim, a plugin to implement Org-roam in neovim.
Scroll further down to see details about the plugin, code pointers and highlights, etc.
The original bug report is kept below for clarity.
Describe the bug
I am writing a plugin that creates some buffers with a buftype of
nofile. The purpose of these is to generate some immutable orgmode content that you can navigate. In particular, I want to take advantage of the fantastic folding that orgmode offers.Unfortunately, when I go to collapse a section, I get an error about the current file is not found or not an org file, caused by
orgmode/lua/orgmode/files/init.lua
Line 106 in 261c987
Steps to reproduce
vim.api.nvim_create_buf(false, true)vim.api.nvim_buf_set_option(bufnr, "filetype", "org")vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, {"* Heading 1-a", "** Heading 2", "* Heading 1-b"})Expected behavior
Section is properly folded.
Emacs functionality
No response
Minimal init.lua
-- Import orgmode using the minimal init exampleScreenshots and recordings
No response
OS / Distro
MacOS 14.4
Neovim version/commit
0.9.5
Additional context
Using orgmode on master branch at commit 651078a.
The text was updated successfully, but these errors were encountered: