Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add local mode for SQLFlow #2997

Open
lhw362950217 opened this issue Oct 9, 2020 · 2 comments
Open

Add local mode for SQLFlow #2997

lhw362950217 opened this issue Oct 9, 2020 · 2 comments
Labels
design Refactory

Comments

@lhw362950217
Copy link
Collaborator

lhw362950217 commented Oct 9, 2020 •
edited
Loading

As discussed before, we'd better add a local mode for SQLFlow, because:

  1. users may want a more light weight installation way, for just try out SQLFlow on her/his PC
  2. developer may want a simpler way to debug the compiler and the generated code

Some thought about the local mode:

  1. We can exclude Kubernetes from the framework, just generate a python file to run docker at local
  2. We need to provide a mechanism to get track of the jobs and their log like in workflow mode
    • user submit a SQL program from Jupyter Notebook/CLI
    • SQLFlow server compile the program and generate a python file
    • SQLFlow server starts a process to run the python file
    • SQLFlow server returns a process id or a log file name as an identifier of the current task(the same way as workflow mode)
    • client poll the log using the identifier returned by SQLFlow server
  3. If we go to an extreme of the local mode, we may only provide a compiler binary, which generate a python file, users can run the python code to do ml tasks, like discussed in this issue
@brightcoder01
Copy link
Collaborator

  1. We can exclude Kubernetes from the framework, just generate a python file to run docker at local

Do we need Minikube here?

  1. We need to provide a mechanism to get track of the jobs and their log like in workflow mode

    • user submit a SQL program from Jupyter Notebook/CLI

    • SQLFlow server compile the program and generate a python file

    • SQLFlow server starts a process to run the python file

    • SQLFlow server returns a process id or a log file name as an identifier of the current task(the same way as workflow mode)

    • client poll the log using the identifier returned by SQLFlow server

We need a sqlflowserver here, I believe it runs in a pod in Minikube.

@lhw362950217
Copy link
Collaborator Author

  1. We can exclude Kubernetes from the framework, just generate a python file to run docker at local

Do we need Minikube here?

  1. We need to provide a mechanism to get track of the jobs and their log like in workflow mode

    • user submit a SQL program from Jupyter Notebook/CLI
    • SQLFlow server compile the program and generate a python file
    • SQLFlow server starts a process to run the python file
    • SQLFlow server returns a process id or a log file name as an identifier of the current task(the same way as workflow mode)
    • client poll the log using the identifier returned by SQLFlow server

We need a sqlflowserver here, I believe it runs in a pod in Minikube.

SQLFlow server can run directly on a native machine, or in a Docker container. So, we may get rid of Minikube.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
design Refactory
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lhw362950217 @brightcoder01