Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New plugin: AI Image Generator #5378

Open
2 tasks done
nqst opened this issue Jul 30, 2024 · 5 comments
Open
2 tasks done

New plugin: AI Image Generator #5378

nqst opened this issue Jul 30, 2024 · 5 comments
Assignees
Labels

Comments

@nqst
Copy link
Contributor

nqst commented Jul 30, 2024

Initial checklist

  • I understand this is a feature request and questions should be posted in the Community Forum
  • I searched issues and couldn’t find anything (or linked relevant results below)

Problem

We want to enhance Uppy's functionality by adding AI features. The first candidates are AI image generation and background removal.

Solution

Here are the screenshots of the new AI image generator feature. This includes the updated Image editor (see #5407), which looks much lighter, and adds the "Edit with AI" and "Remove background" features.

  1. The Dashboard with the new "AI Image Generator" plugin:
  1. Form. It's very similar to the form that we currently have in the Unsplash plugin:
  1. Result images. The idea is to generate 2 images by default (this amount is configurable) so the user can select their preferred option. The selection logic is similar to what we have in the Instagram plugin — clicking on an image toggles a bar with the Select button.
  1. Updated Image Editor. (Update the Image Editor UI #5407) It contains the new AI features. The "Edit with AI" button toggles a form (see Screenshot 5 below), while the "Remove bg" is a button which immediately triggers the automatic background removal process.
  1. Edit with AI mode active.

These screenshots are likely not final, and the mobile version is not ready yet. Comments and suggestions are welcome!

@nqst nqst added the Feature label Jul 30, 2024
@nqst nqst self-assigned this Jul 30, 2024
@Murderlon
Copy link
Member

@nqst there are currently no loading and error states. What happens when "Edit with AI" or Remove "bg" take 20 seconds? Or fail? Can we add some designs for that?

@nqst
Copy link
Contributor Author

nqst commented Aug 6, 2024

What happens when "Edit with AI" or Remove "bg" take 20 seconds? Or fail? Can we add some designs for that?

Absolutely! I'll add these.

@Murderlon
Copy link
Member

We should also consider splitting this issue into two: image editor changes and the new plugin. They can be worked on separately and are quite different things to build.

@nqst nqst changed the title AI Image Generator Updated Image Editor with prompt edit New plugin: AI Image Generator Aug 14, 2024
@Murderlon Murderlon assigned Murderlon and unassigned aduh95 Aug 19, 2024
@kvz
Copy link
Member

kvz commented Sep 2, 2024

Even if https://github.com/transloadit/team-internals/issues/339 isn't finished yet, we could already leverage Replicate.com/Fal.ai as suggested here.

And then use FLUX, while we don't support it yet.

This seemed to be hard, what's the status of this Merlijn? Tim and I would like to push for this

@Murderlon
Copy link
Member

By only relying on @uppy/transloadit and the Robot, instead of leveraging third-party APIs, this plugin can now be very simple.

It's 80% done with just 50 lines of code.

Screen.Recording.2024-09-02.at.11.21.20.mov
  search = async () => {
    const transloadit = this.uppy.getPlugin<Transloadit<M, B>>('Transloadit')

    if (!transloadit) {
      throw new Error('ImageGenerator requires the Transloadit plugin')
    }

    this.uppy.on('transloadit:result', (stepName, result) => {
      // TODO: use a deterministic property to get the right result,
      // not stepName which can be changed by the implementer of the template. 
      if (stepName === 'resized') {
        const { results } = this.getPluginState()
        this.setPluginState({ results: [...results, result] })
      }
    })

    try {
      await this.uppy.upload()
    } finally {
      // Normally uppy.upload() is only used to upload your files
      // but here we need it to _get_ the results from the AI image generator Robot.
      // That means users who set `allowMultipleUploadBatches: false` will not
      // be able to actually upload their files, so we reset the state here.
      this.uppy.setState({ allowNewUpload: true })
    }
  }

  private onCheckboxChange = (result: AssemblyResult) => {
    const { checkedResultIds } = this.getPluginState()

    if (checkedResultIds.has(result.id)) {
      checkedResultIds.delete(result.id)
    } else {
      checkedResultIds.add(result.id)
    }

    this.setPluginState({ checkedResultIds })
  }

  private donePicking = () => {
    const { checkedResultIds, results } = this.getPluginState()
    const files = results
      .filter((result) => checkedResultIds.has(result.id))
      .map((result) => ({
        name: result.name,
        data: { size: result.size },
      }))

    this.uppy.addFiles(files)
  }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants