Emacs hangs for 10 sec on asynchronous execution

[stuartthomas25]

When I run a block of asynchronous code, Emacs hangs for 10 seconds if the total execution takes longer than 10 seconds.

I managed to track this behavior down to the line

(ob-julia-vterm-wait-for-file-change .out-file 10 0.1)

in the function ob-julia-vterm-evaluation-completed-callback-func . It looks like ob-julia-vterm is waiting for the output file to change and has a timeout of 10 seconds. I'm not sure why this is called here. I'm also not sure why the output file only displays the final result instead of stdout since ob-julia-vterm-make-str-to-run seems to pipe stdout to this file.

I tried commenting out this line and it seems to fix the issue, but maybe I'm missing why it's there in the first place.

[shg]

Hi @stuartthomas25

Thank you for trying out ob-julia-vterm and for the report.
I would like to have a bit more information about what you executed and what happened. On my machine, if I write the following three blocks in a file and execute them at once with org-babel-execute-buffer, the result of the last block has a value of just over 12 seconds. Do you get a different result?

#+BEGIN_SRC julia :results none
using Dates
t0 = now()
#+END_SRC

#+BEGIN_SRC julia :results none
sleep(12)
#+END_SRC

#+BEGIN_SRC julia :results value
now() - t0
#+END_SRC

#+RESULTS:
: 12436 milliseconds

[shg]

Regarding your second question, I guess you are talking about how the results are collected by Emacs Org Babel. And what you see is the expected behavior. To collect outputs to the stdout, you have to specify output for the results header argument. The default collection mode is value, in which Babel gets the result of final expression in the code block.

[stuartthomas25]

Hi @shg,

Thanks for your reply.

Regarding your first point, the issue I'm experiencing is not with the execution but with Emacs freezing. While the code you provided is running, I am not able to move the cursor or write any text. Afterwards, it returns the same value. Emacs is waiting synchronously for 10 seconds, though the actual execution is not affected. For your second question, I am talking about the output file, not the actual result. This is the file stored in the /tmp/babel directory.

The callback function in this repo waits until the output file is no longer empty using the function ob-julia-vterm-wait-for-file-change, but I'm realizing this fails if the results header argument is value or silent or furthermore if stdout is not flushed.

[shg]

Hi @stuartthomas25

I would like to see exactly what you execute, including code and header, could you provide one?
Also based on your explanation I understand the issue does not always happen but happens only for the blocks that take longer than 10 sec. That is, Emacs doesn't hang for the block that takes 9 sec?

Thank you!

[stuartthomas25]

Hi @shg,

You can reproduce this with just

#+BEGIN_SRC julia
sleep(15)
#+END_SRC

Emacs hangs until execution is finished or until 10 seconds have elapsed. I attached a video

Screencast.from.2024-05-04.20-17-12.mp4

[shg]

Thank you for the video. That helps. So in your environment, Emacs basically always hangs, no matter how long the block's execution time is, but up to 10 sec. Is my understanding correct?

The callback function should be called when a block's execution is completed. That ob-julia-vterm-wait-for-file-change call is there just to make sure that the results are saved to a temporary output file. It performs synchronous waiting but it doesn't matter since normally it returns immediately. But in your environment, it seems that the callback function is called at the start of block execution thus that waiting time.

Can you give me information on your operating system and environment?

[stuartthomas25]

I now see this is an issue on my end, I'm not using the most up-to-date version. Sorry for wasting your time.

Let me know if you need any help with this project though! I use it nearly every day for work.

[shg]

Oh, that's good to hear. No worries at all. I'm glad to hear that the problem is resolved!

I use vanilla Emacs, and I haven't really tested this package in other environments much. So feedback from Doom users like you is very helpful. I appreciate your issue reports and suggestions.

[stuartthomas25]

So it looks like this is happening again on the most up-to-date version with the same behavior. The .out-file is only written to at the end of execution so ob-julia-vterm-wait-for-file-change hangs Emacs until this point.

[shg]

@stuartthomas25
Thank you for the report.

Could you please provide more specific information about your environment, what you executed, and what you experienced?

Additionally, I released the latest updates for this package and julia-vterm.el on May 14, 2024, and there have been no updates since then. Did the issue start occurring immediately after you updated to these latest versions?

[stuartthomas25]

Ok, I went back to see which commit started having the problem and it seems to start at 354de54, which is the first to add the ob-julia-vterm-wait-for-file-change function. I'm not sure why this issue went away in a couple months ago.

[shg]

Could you please provide more specific information about your environment, what you executed, and what you experienced?

[stuartthomas25]

The behavior is as described before. I am on Emacs 29.4 on Fedora 40, Linux 6.9.7-200, x86_64.

Looking through the code it seems clear to me that ob-julia-vterm-wait-for-file-change is waiting for the output file (.out-file) to change, which it does not do until the end of execution. I guess what I don't understand is if ob-julia-vterm is supposed to write to this file at the beginning of execution. Why does ob-julia-vterm wait for the file change at all?

[shg]

Hi @stuartthomas25

Why does ob-julia-vterm wait for the file change at all?

It does that to ensure that the results are stored in the temporary file. This check is performed after the block execution has finished. So if everything is working as intended, the ob-julia-vterm-wait-for-file-change function should not block Emacs because it is only called from a callback function that is called when the block execution has completed.

A possiblity may be that the callback function is being called earlier with some unknown (maybe environment specific?) reason.

I am testing it with Emacs 29.4 on my Linux machine but haven't reproduced the issue yet.

Are you still using Doom? Also I appreciate it if you can provide information about any other non-standard things in your environment if any.

Also it helps if you can test whether the issue occurs with a bare Emacs in your environment.

[stuartthomas25]

Here's some more details:

My Doom config:

generated  Jul 25, 2024 10:40:21
system     "Fedora release 40 (Forty)" Linux 6.9.7-200.fc40.x86_64 x86_64
emacs      29.4 EMACSDIR=~/.emacs.d/ EMACS=/usr/bin/emacs
doom       3.0.0-pre PROFILE=_@0 HEAD -> master 517daa4e 2024-06-03 16:22:10 -0400 ~/.config/doom/
shell      /usr/bin/zsh
features   ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 MODULES
	   NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER
	   WEBP XIM XWIDGETS GTK3 ZLIB
traits     batch server-running envvar-file
modules    :config use-package :completion company (vertico +icons) :ui doom (emoji +unicode) hl-todo ligatures modeline ophints
	   (popup +defaults) unicode vc-gutter vi-tilde-fringe workspaces :editor (evil +everywhere) file-templates fold
	   multiple-cursors snippets word-wrap :emacs (dired +icons) electric (ibuffer +icons) undo vc :term term :checkers
	   syntax (spell +flyspell) :tools (eval +overlay) lookup lsp magit make pdf rgb upload :lang cc emacs-lisp ess (haskell
	   +dante +lsp) (julia +lsp +snail) (latex +lsp +latexmk +fold) markdown ocaml (org +roam2 +pretty +dragndrop +jupyter)
	   (python +conda +lsp) qt sh :email (mu4e +org +gmail) :app calendar (rss +org) twitter :config literate (default
	   +bindings +smartparens)
packages   (dashboard) (helm-bibtex) (org-ref) (julia-vterm :pin 6d67b61) (ob-julia-vterm :pin 354de54) (org-auto-tangle)
	   (wolfram-mode) (ob-mathematica) (org-modern) (org-inline-pdf) (org-roam-server) (julia-quail :recipe (:host github
	   :repo stuartthomas25/julia-quail)) (polymode) (arxiv-mode) (calibredb) (slurm-mode) (lean4-mode :recipe (:host github
	   :repo leanprover/lean4-mode :files (*.el data)))

and Julia Version 1.10.3 (2024-04-30)

[stuartthomas25]

So I think I got to the root cause of the issue. When ob-julia-vterm pastes the src code into the REPL, it immediately opens the out-file. On my system for some reason, this causes an inotify "MODIFY" event, which then prompts the callback. The simple fix here is to open the out-file in append mode. I just filed pull request #28 that makes this change. I'm still not sure why this issue went away but it seems very dependent on Julia's behavior, specifically its interaction with the OS.

[shg]

@stuartthomas25

Thank you for the investigation and PR.I was also checking the behaviors of filenotify on the file written by Julia. Give me some time to examine the possible implications of the changes.