Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug: Model generates response non-stop (default prompt template is wrong?) #3584

Closed
2 of 4 tasks
imtuyethan opened this issue Sep 6, 2024 · 6 comments
Closed
2 of 4 tasks
Assignees
Labels
category: threads & chat Threads & chat UI UX issues type: bug Something isn't working
Milestone

Comments

@imtuyethan
Copy link
Contributor

imtuyethan commented Sep 6, 2024

  • I have searched the existing issues

Current behavior

https://discord.com/channels/1107178041848909847/1281512188762128467/1281537450111406080
https://discord.com/channels/1107178041848909847/1239846009258119178/1281127913525088318
https://discord.com/channels/1107178041848909847/1107178593945129060/1281169582811250719

  1. Models downloaded from the Jan Hub (including Llama 3) have incorrect default settings:
    • Fixed context length of 2048
    • Incorrect prompt template
    • Incorrect stop word
  2. The Llama 3 model is repeating responses, producing duplicate or very similar output multiple times.
  3. When importing GGUF models directly (not through the hub), settings appear correct.

Minimum reproduction step

  1. Download a GGUF model (e.g., Llama 3) from the Jan Hub
  2. Observe incorrect default settings
  3. Start a new conversation thread
  4. Send a prompt to the model
  5. Observe the repeated responses and incorrect formatting
  6. Try importing the same GGUF model directly and compare settings

Expected behavior

  1. Models downloaded from the Jan Hub should have correct default settings.
  2. The Llama 3 model (and other models) should provide a single, coherent response without repetition.
  3. The correct prompt template should be used for proper interaction.
  4. Directly imported models and hub-downloaded models should have consistent, correct settings.

Screenshots / Logs

Screen.Recording.2024-09-06.at.3.37.18.PM.mov

Screenshot 2024-09-06 at 3 54 08 PM

Screenshot 2024-09-06 at 3 58 24 PM

Jan version

v0.5.3-623

In which operating systems have you tested?

  • macOS
  • Windows
  • Linux

Environment details

Operating System: MacOS Sonoma 14.2
Processor: Apple M2
RAM: 16GB

@imtuyethan imtuyethan added type: bug Something isn't working P0: critical Mission critical labels Sep 6, 2024
@imtuyethan imtuyethan changed the title bug: Model generates response non-stop bug: Model generates response non-stop (default prompt template is wrong?) Sep 6, 2024
@louis-jan
Copy link
Contributor

louis-jan commented Sep 9, 2024

Findings:

Ashley had already removed some of the models, and the old threads lost their model references. When she opens the thread, it automatically selects another available model, which has a legacy of bad UX.

There is another issue created to address that (searching for it...)

@0xSage
Copy link
Contributor

0xSage commented Sep 9, 2024

@louis-jan @imtuyethan I dont understand the issue, can you guys explain a bit more clearly?

is this an edge case where:

  1. Old thread
  2. Model used for that thread is deleted (along with model.json)
  3. User sends more messages in old thread
  4. Thread autoselects some other model
  5. New model has corrupted settings? How?

@tikikun
Copy link
Contributor

tikikun commented Sep 16, 2024

Should be able to extract chat template from gguf, not good ux

@louis-jan
Copy link
Contributor

@louis-jan @imtuyethan I dont understand the issue, can you guys explain a bit more clearly?

is this an edge case where:

  1. Old thread
  2. Model used for that thread is deleted (along with model.json)
  3. User sends more messages in old thread
  4. Thread autoselects some other model
  5. New model has corrupted settings? How?

Hi @0xSage @tikikun. This is an outdated issue, it should link to the updated one. This is the case where the model is deleted and the app selects another available model to continue the conversation. However, that is an old behavior that we will update soon.

@imtuyethan please help link the new issue.

@imtuyethan imtuyethan added this to the v0.5.5 milestone Sep 18, 2024
@imtuyethan
Copy link
Contributor Author

imtuyethan commented Sep 18, 2024

There are two issues related to this:

  1. For this one: https://discord.com/channels/1107178041848909847/1281512188762128467/1281537450111406080, it's because I deleted model A, threads defaulted to model B & the settings were not updated, so threads used model B but settings from model A, causing the weird responses (wrong prompt template, etc.,...). This should be fixed via feat: Pick new model in thread if original model is unavailable #3385.

  2. bug: Dropping GGUF metadata when creating model.yaml for Model Downloaded via URL #3558

@imtuyethan imtuyethan added category: model running and removed P0: critical Mission critical labels Sep 18, 2024
@imtuyethan imtuyethan modified the milestones: v0.5.5, v0.5.6 Sep 23, 2024
@0xSage 0xSage added category: threads & chat Threads & chat UI UX issues and removed category: engines labels Oct 14, 2024
@0xSage
Copy link
Contributor

0xSage commented Oct 15, 2024

dupe of #3385

@0xSage 0xSage closed this as completed Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: threads & chat Threads & chat UI UX issues type: bug Something isn't working
Projects
Archived in project
Development

No branches or pull requests

4 participants