Skip to content

Pre-configured AI server // How? #692

Closed
Closed
Pre-configured AI server // How?#692
Labels
questionFurther information is requested

Description

@UberMetroid
UberMetroid
opened on Oct 26, 2024

Bug description

On your demo site, you have a pre-configured AI server.
Screenshot 2024-10-26 072149

How did you set this up?

Steps to reproduce

Here are my current docker settings:
Screenshot 2024-10-26 072356

Expected behavior

When I load my site https://search.robocanvas.io it should auto load with my pre-configured ollama server AND the user should not have the option to change the configuration.

Device info

No response

Additional context

No response

Activity

  1. UberMetroid
    added
    bugSomething isn't working
    on Oct 26, 2024

  2. felladrin commented on Oct 26, 2024

    @felladrin
    felladrin
    on Oct 26, 2024
    Owner

    Ah, I see! To use the environment variables from the OS, you need to pass the environment variables to the container manually.

    So if you're running docker run -p 7860:7860 ghcr.io/felladrin/minisearch:main, pass all the env vars to the command, like this:

    docker run -p 7860:7860 -e INTERNAL_OPENAI_COMPATIBLE_API_BASE_URL=$INTERNAL_OPENAI_COMPATIBLE_API_BASE_URL -e INTERNAL_OPENAI_COMPATIBLE_API_KEY =$INTERNAL_OPENAI_COMPATIBLE_API_KEY -e INTERNAL_OPENAI_COMPATIBLE_API_MODEL=$INTERNAL_OPENAI_COMPATIBLE_API_MODEL -e INTERNAL_OPENAI_COMPATIBLE_API_NAME=$INTERNAL_OPENAI_COMPATIBLE_API_NAME ghcr.io/felladrin/minisearch:main

    But if you are using docker compose -f docker-compose.production.yml up --build, it means you have access to the machine and have cloned the repository there. In this case, it's preferable to create a .env and set the variables in it. Is it possible? (There's an alternative, but I'll only comment on it if you say it's not possible in your current environment)

    Let me know if it helps.

  3. UberMetroid commented on Oct 26, 2024

    @UberMetroid
    UberMetroid
    on Oct 26, 2024
    Author

    I think that is what I have been asking, in regard to mounting the volumes. that way I can create and edit the .env.

    I have full control of the machine.

  4. felladrin commented on Oct 26, 2024

    @felladrin
    felladrin
    on Oct 26, 2024
    Owner

    Ah ok, then I need to clarify it in the Readme.

    Please try this:

    git clone https://github.com/felladrin/MiniSearch.git
cd MiniSearch
cp .env.example .env

    Edit the .env, then run:

    docker compose -f docker-compose.production.yml up --build

    This will build and run the docker container, already taking all the files in MiniSearch into account, including the .env.

    And to update it to the latest version, stop the container, and run:

    git pull
docker compose -f docker-compose.production.yml up --build

    P.S. I know you know the commands/sequence already, but I'm just writing in detail so I can reference this message if other issues are open about this later.

  5. UberMetroid commented on Oct 26, 2024

    @UberMetroid
    UberMetroid
    on Oct 26, 2024
    Author

    I don't use those commands for my containers. I'm stuck with a GUI for building and maintaining containers (unRaid OS). Are there any flags to turn off the browser or openai interface?

    Also, point of confusion, the AI Processing Location has default options of
    In the browser (Private)
    Remote Server (API)

    What option leads to what DEFAULT_INFERENCE_TYPE ? Consider naming them the same thing and graying out options that are not available.

  6. felladrin commented on Oct 26, 2024

    @felladrin
    felladrin
    on Oct 26, 2024
    Owner

    DEFAULT_INFERENCE_TYPE="internal", as you set, is correct: means whenever a new user visits the app for the first time, this will be the option selected by default. (You can confirm it by visiting the app in a private browser window). (On the first visit, the settings are saved on the localStorage, so it's really only used on the user's first visit).

    Ah yes, about the confusion, it's because I've recently renamed those options, but I forgot to update the .env.example to have those new wording.

    But here's how they map:

    • "browser" -> In the browser (Private)
    • "openai" -> Remote Server (API)
    • "internal" -> INTERNAL_OPENAI_COMPATIBLE_API_NAME

    Are there any flags to turn off the browser or openai interface?

    No, as this would go against the main purpose of the app, which was to run models directly from the browser (the OpenAI-Compatible API was only added recently). Could you tell me more about why you think there should be this option?

    But going back to the topic, if unRaid OS runs docker build . (instead of running the commands mentioned before) and there's a .env there, it will also work.

    Please let me know if any of the solutions worked.

  7. felladrin
    added a commit that references this issue on Oct 26, 2024

    Improve description of the `DEFAULT_INFERENCE_TYPE` variable

    0223631

  8. UberMetroid commented on Oct 26, 2024

    @UberMetroid
    UberMetroid
    on Oct 26, 2024
    Author

    No, as this would go against the main purpose of the app, which was to run models directly from the browser (the OpenAI-Compatible API was only added recently). Could you tell me more about why you think there should be this option?

    I'm not using the app with browser based LLMs. Ill use it with my ollama server for private search. I view it is a very lite weight webui for llms.

  9. felladrin commented on Oct 26, 2024

    @felladrin
    felladrin
    on Oct 26, 2024
    Owner

    I'm not using the app with browser based LLMs. Ill use it with my ollama server for private search. I view it is a very lite weight webui for llms.

    Ah, so it's to avoid loading the libraries that run the LLMs in the browser, making the app lighter, right?

    No need to worry, those libraries are only loaded if you have the "In the browser (Private)" AI Processing Location selected. It was all modularized to use only what users needed.

    By the way, this discussion gave me great insights, so I thought about simplifying the server configuration, which is becoming complicated with all those environment variables:

  10. felladrin commented on Oct 26, 2024

    @felladrin
    felladrin
    on Oct 26, 2024
    Owner

    Back to the original question, were you able to make it display the custom option to use the internal API?
    image

  11. UberMetroid commented on Oct 26, 2024

    @UberMetroid
    UberMetroid
    on Oct 26, 2024
    Author

    Back to the original question, were you able to make it display the custom option to use the internal API? image

    still no. I did figure out how to copy and paste the command into the command line to start the container. so thats a start.

  12. felladrin
    added
    questionFurther information is requested
    and removed
    bugSomething isn't working
    on Oct 27, 2024

  13. UberMetroid commented on Oct 29, 2024

    @UberMetroid
    UberMetroid
    on Oct 29, 2024
    Author

    Still not working. I'm stuck.

  14. felladrin commented on Oct 29, 2024

    @felladrin
    felladrin
    on Oct 29, 2024
    Owner

    I’m out of ideas on how to address this issue.

    In the meantime, as you search for a solution to get the Internal API working, can you use the "Remote Server (API)" option to connect to Ollama directly from the browser?

  15. UberMetroid commented on Oct 29, 2024

    @UberMetroid
    UberMetroid
    on Oct 29, 2024
    Author

    I’m out of ideas on how to address this issue.

    In the meantime, as you search for a solution to get the Internal API working, can you use the "Remote Server (API)" option to connect to Ollama directly from the browser?

    I can, and thats what I currently have set up. Works great.

  16. felladrin commented on Oct 30, 2024

    @felladrin
    felladrin
    on Oct 30, 2024
    Owner

    That's good! Then, I'm converting this issue into a discussion to keep it open in case more information related to this arises.

  17. Repository owner locked and limited conversation to collaborators on Oct 30, 2024
  18. felladrin
    converted this issue into a discussion #706 on Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions