Expected value but found invalid token at character 1

[gammons]

Hi there, I am trying to use CopilotChat.nvim. I think maybe the main difference is I am using packer.nvim as my package manager. I am installing Copilot chat like so:

  use {
    'CopilotC-Nvim/CopilotChat.nvim',
    branch = 'canary',
    requires = {
      'github/copilot.vim',
      'nvim-lua/plenary.nvim',
    }
  }

-- copilot chat
require("CopilotChat").setup({
   debug = true,
})

Then I run PackerSync and it looks like CopilotChat is installed correctly. (PackerSync is also making sure that all other libraries / dependencies are up-to-date as well)

Here is the output of checkhealth CopilotChat:

CopilotChat: require("CopilotChat.health").check()

CopilotChat.nvim [core] ~
- OK nvim: NVIM v0.10.1
  Build type: Release
  LuaJIT 2.1.1725453128
  Run ":verbose version" for more info

CopilotChat.nvim [commands] ~
- OK curl: curl 8.9.1 (x86_64-pc-linux-gnu) libcurl/8.9.1 OpenSSL/3.3.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.63.0 nghttp3/1.5.0
  Release-Date: 2024-07-31
  Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
  Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd
- OK git: git version 2.46.0

CopilotChat.nvim [dependencies] ~
- OK plenary: installed
- OK copilot: copilot.vim
- WARNING tiktoken_core: missing, optional for token counting. See README for installation instructions.
- OK treesitter[markdown]: installed
- OK treesitter[diff]: installed

However, if I try to enter a prompt into the window, or if I run CopilotChatExplain, I am getting the following error:

[CopilotChat.nvim] "Expected value but found invalid token at character 1

Any help to debug would be greatly appreciated. Thanks!

[thomastthai]

This error started happening on my setup today too. I use the Lazy plugin manager and this has been working fine for months until today. It occurs when executing commands like:

:'<,'>CopilotChat check for errors

[deathbeam]

I improved the error logging so can you check again now and check the error? @thomastthai @gammons

[thomastthai]

@deathbeam, thank you for adding additional error logging.

Following up on the info I posted from #436:

1. Update CopilotChat via Lazy
2. Quit all from nvim
3. Reload the same file via nvim
4. Got this new error (also cited in Vimscript function must not be called in a lua loop callback #444):

Error executing luv callback:
.../nvim/lazy/CopilotChat.nvim/lua/CopilotChat/tiktoken.lua:8: E5560: Vimscript function must not be called in a lua loop callback
stack traceback:
        [C]: in function 'mkdir'
        .../nvim/lazy/CopilotChat.nvim/lua/CopilotChat/tiktoken.lua:8: in function 'get_cache_path'
        .../nvim/lazy/CopilotChat.nvim/lua/CopilotChat/tiktoken.lua:32: in function <.../nvim/lazy/CopilotChat.nvim/lua/CopilotChat/tiktoken.lua:30>
        [C]: in function 'nvim_exec2'
        vim/_editor.lua: in function 'cmd'
        ...local/share/nvim/lazy/lazy.nvim/lua/lazy/core/loader.lua:506: in function <...local/share/nvim/lazy/lazy.nvim/lua/lazy/core/loader.lua:505>
        [C]: in function 'xpcall'
        .../.local/share/nvim/lazy/lazy.nvim/lua/lazy/core/util.lua:135: in function 'try'
        ...local/share/nvim/lazy/lazy.nvim/lua/lazy/core/loader.lua:505: in function 'source'
        ...local/share/nvim/lazy/lazy.nvim/lua/lazy/core/loader.lua:453: in function 'source_runtime'
        ...local/share/nvim/lazy/lazy.nvim/lua/lazy/core/loader.lua:421: in function 'packadd'
        ...local/share/nvim/lazy/lazy.nvim/lua/lazy/core/loader.lua:139: in function 'startup'
        ...homas/.local/share/nvim/lazy/lazy.nvim/lua/lazy/init.lua:112: in function 'setup'
        /Users/thomas/.config/nvim/lua/thomasthai/lazy.lua:14: in main chunk
        [C]: in function 'require'
        /Users/thomas/.config/nvim/init.lua:71: in main chunk

After acknowledging the error by pressing ENTER, I visual-select the entire buffer and executed the nvim command: :'<,'>CopilotChatFix check errors, nvim shows the error, [CopilotChat.nvim] Unprocessable Entity.

Here is the tail of CopilotChat.nvim.log.

.local/state/nvim
❯ tail -400 CopilotChat.nvim.log | pbcopy

  }

  @override
  Widget build(BuildContext context) {
    return Row(
      children: [

[... Flutter code deleted for brevity ...]

      TaskNode(
        id: Editor.createNodeId(),
        isComplete: true,
        text: AttributedText(
          'This is a completed task',
        ),
      ),
    ],
  );
}
[DEBUG Wed Oct 30 00:01:52 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:422: Model: gpt-4o-2024-05-13
[DEBUG Wed Oct 30 00:01:52 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:423: Temperature: 0.1
[ERROR Wed Oct 30 00:01:54 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:522: Failed parse response: 
Unprocessable Entity
"Expected value but found invalid token at character 1"
[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:442: System prompt: You are an AI programming assistant.
When asked for your name, you must respond with "GitHub Copilot".
Follow the user's requirements carefully & to the letter.
Follow Microsoft content policies.
Avoid content that violates copyrights.
If you are asked to generate content that is harmful, hateful, racist, sexist, lewd, violent, or completely irrelevant to software engineering, only respond with "Sorry, I can't assist with that."
Keep your answers short and impersonal.
You can answer general programming questions and perform the following tasks: 
* Ask a question about the files in your current workspace
* Explain how the code in your active editor works
* Generate unit tests for the selected code
* Propose a fix for the problems in the selected code
* Scaffold code for a new workspace
* Create a new Jupyter Notebook
* Find relevant code to your query
* Propose a fix for the a test failure
* Ask questions about Neovim
* Generate query parameters for workspace search
* Ask how to do something in the terminal
* Explain what just happened in the terminal
You use the GPT-4 version of OpenAI's GPT models.
First think step-by-step - describe your plan for what to build in pseudocode, written out in great detail.
Then output the code in a single code block. This code block should not contain line numbers (line numbers are not necessary for the code to be understood, they are in format number: at beginning of lines).
Minimize any other prose.
Use Markdown formatting in your answers.
Make sure to include the programming language name at the start of the Markdown code blocks.
Avoid wrapping the whole response in triple backticks.
The user works in an IDE called Neovim which has a concept for editors with open files, integrated unit test support, an output pane that shows the output of running the code as well as an integrated terminal.
The user is working on a Darwin machine. Please respond with system specific commands if applicable.
The active document is the source code the user is looking at right now.
You can only give one reply for each conversation turn.
You also specialize in being a highly skilled code generator. Given a description of what to do you can refactor, modify, enhance existing code or generate new code. Your task is help the Developer change their code according to their needs. Pay especially close attention to the selection context.

Additional Rules:
Markdown code blocks are used to denote code.
If context is provided, try to match the style of the provided code as best as possible. This includes whitespace around the code, at beginning of lines, indentation, and comments.
Preserve user's code comment blocks, do not exclude them when refactoring code.
Your code output should keep the same whitespace around the code as the user's code.
Your code output should keep the same level of indentation as the user's code.
You MUST add whitespace in the beginning of each line in code output as needed to match the user's code.
Your code output is used for replacing user's code with it so following the above rules is absolutely necessary.

[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:443: Prompt: There is a problem in this code. Rewrite the code to show it with the bug fixed. check errors
[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:444: Embeddings: 0
[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:445: Filename: /Volumes/Dev/Programming/Flutter/super_editor_mix_mode/lib/main.dart
[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:446: Filetype: dart
[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:447: Selection: import 'dart:async';
import 'package:flutter/material.dart';
import 'package:super_editor/super_editor.dart';
import 'package:super_editor_markdown/super_editor_markdown.dart';

void main() {
  runApp(MarkdownSerializationDemo());
}


[... Flutter code deleted for brevity ...]

      TaskNode(
        id: Editor.createNodeId(),
        isComplete: true,
        text: AttributedText(
          'This is a completed task',
        ),
      ),
    ],
  );
}
[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:448: Model: gpt-4o-2024-05-13
[DEBUG Wed Oct 30 08:34:16 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:449: Temperature: 0.1
[ERROR Wed Oct 30 08:34:18 2024] /Users/thomas/.local/share/nvim/lazy/CopilotChat.nvim/lua/CopilotChat/copilot.lua:545: Failed to parse response: "Expected value but found invalid token at character 1"
Unprocessable Entity

[gammons]

FYI, I was able to get copilot chat to work! I did not have to do anything other than run PackerSync again, and it seems to have fixed itself.

One note, I did have to enable "Copilot chat in the IDE" to get this to work. (this is at https://github.com/settings/copilot)

Probably it should be called out in the README that this setting needs to be enabled.

[gammons]

@thomastthai FWIW I was getting that error as well. For me it was happening because I had luajit-tiktoken-bin installed (I'm using Arch linux). When I removed that library, copilot chat worked for me.

[gammons]

going to close this issue since this seems to be fixed for me.

[thomastthai]

@gammons, I am glad your setup is working now!

One note, I did have to enable "Copilot chat in the IDE" to get this to work. (this is at https://github.com/settings/copilot)

My "Copilot chat in the IDE" was enabled per #436 when this error cropped up.

[thomastthai]

@gammons, I am glad your setup is working now!

Please keep the issue open as @deathbeam merged #436 with this issue saying it was a duplicate.

[thomastthai]

@thomastthai FWIW I was getting that error as well. For me it was happening because I had luajit-tiktoken-bin installed (I'm using Arch linux). When I removed that library, copilot chat worked for me.

README.md listed luajit-tiktoken as an option:

Optional:

tiktoken_core: sudo luarocks install --lua-version 5.1 tiktoken_core. Alternatively, download a pre-built binary from lua-tiktoken releases
You can check your Lua PATH in Neovim by doing :lua print(package.cpath). Save the binary as tiktoken_core.so in any of the given paths.
For Arch Linux user, you can install luajit-tiktoken-bin or lua51-tiktoken-bin from aur!

If a conflict arises between CopilotChat and luajit-tiktoken, wouldn't fixing the root issue be better than removing luajit-tiktoken, as other plugins or packages may depend on it?

@gammons, I'm on MacOS and have luajit and tiktoken_core installed. I can't remember which other extension(s) or package(s) depend on them. They were installed when CopilotChat was working.

❯ brew list | grep lua
lua
luajit
luarocks

❯ luarocks list

Rocks installed for Lua 5.4
---------------------------

luarocks-build-rust-mlua
   0.2.2-1 (installed) - /Users/thomas/.luarocks/lib/luarocks/rocks-5.4
   0.2.2-1 (installed) - /opt/homebrew/lib/luarocks/rocks-5.4

tiktoken_core
   0.2.2-1 (installed) - /opt/homebrew/lib/luarocks/rocks-5.4

[thomastthai]

Probably it should be called out in the README that this setting needs to be enabled.

@gammons, doesn't "Copilot chat in the IDE" get enabled during the plugin installation? Since there were a few issues mentioning this setting, other users may benefit if README.md should at least give them a heads-up. I submitted a pull request for that change.

[gammons]

@thomastthai I would have thought that as well, but it was disabled for my work org, which is the org that's paying for copilot. The weird thing is that regular copilot autocomplete was working just fine with that IDE option disabled 🤷

[thomastthai]

@gammons, you nailed why that setting was not enabled: work org had it disabled.

[thomastthai]

The commits to the canary branch from 2024-10-30 fixed errors mentioned in this issue, including one related to #444. Feel free to close this issue.