feat(edge-compute): move host jobs to edge #3840

chiptus · 2020-05-20T16:55:33Z

deviantony

Few things I noticed while trying this build:

Cannot start from a clean environment, I get the following error in the Portainer logs:

2020/05/21 06:38:40 [ERROR] [http,client] [message: unexpected status code] [status_code: 403]
2020/05/21 06:38:40 [WARN] [exec,extensions] [message: unable to retrieve extensions manifest via Internet. Extensions will be retrieved from local cache and might not be up to date] [err: Invalid response status (expecting 200)]
2020/05/21 06:38:40 [WARN] [exec,extensions] [message: unable to retrieve extension information from Internet. Skipping extensions update check.] [err: open /extensions.json: no such file or directory]
2020/05/21 06:38:40 time: invalid duration

I believe that the problem here is:

2020/05/21 06:38:40 time: invalid duration

If I then start a 2.0.0-dev instance before trying to run this implementation with the same database, it starts fine.

UI - Settings - Change the tooltip of the Host management switch to:
"Enable host management features: host system browsing and advanced host details." (I just added advanced)

Remove the part about CAP_HOST_MANAGEMENT.

Sidebar looks broken, I only logged in and I can see endpoint level links in the home view.

In the Edge job creation view, when no Edge endpoints are available the selector display a "loading" message. It should display "No endpoint available".

After creating an Edge endpoint, it still displays loading so I can't select any endpoint.

We should prevent the creation of an Edge job if the script file is empty or if there is no associated endpoint

I wasn't able to dig more into this due to the impossibility to create an Edge job at the moment.

chiptus

fixes

app/edge/components/group-form/groupFormController.js

app/edge/services/edge-job.js

deviantony

Review the task panel in the Edge job details view to only keep the tasks datatable

This datatable should only have two columns: Endpoint, Actions

In the Endpoint column, keep the name of the associated endpoint.

In the actions column, introduce a new way to collect logs via the "Retrieve logs" action. When the logs are available (upon refresh of the view), this action should then change to "Download logs".

Introduce support for collecting logs associated to a task

Related agent issue: portainer/agent#135

chiptus · 2020-05-27T10:36:17Z

Introduce a new action that can be used to clear the associated logs for each selected Edge job

I think my suggestion was misunderstood. What I was thinking was to add the batch action to the task table. There we can have collect logs, clear logs. Download logs will be separate for each task.

chiptus · 2020-05-27T12:50:52Z

persist (in DB) which job/endpoint logs are required
send them in endpoint_status_inspect, each job with a collectLogs boolean
add an API route POST /endpoint/:endpoint_id/edge/jobs/:edge_job_id/logs to get logs. this API will cache logs, and set log collection status as received (endpoint should be "edge" secured)
GET /endpoint/:endpoint_id/edge/jobs/:edge_job_id/logs to get the logs (admin access)

suggestion:

replace edgeJob.endpoints from an array of ids to an array of objects {endpointId:int, logsStatus:nil/pending/received}
edgeSchedule (the response object sent to an edge endpoint) - remove endpoints array, I'm pretty sure it's redundant and add collectLogsBoolean based on job.endpoints[endpointId].logsStatus

deviantony · 2020-06-02T23:43:07Z

Please rebase that PR on the develop branch.

pull-dog · 2020-06-03T05:07:38Z

*Ruff* 🐶 The test environment for this pull request has been destroyed 💥 This may have happened explicitly via a command, or because the pull request was closed.

React on this comment to leave anonymous feedback.

👍 to say good dog 🍖
👎 to say bad dog 🦴

Commands

@pull-dog up to reprovision or provision the server.
@pull-dog down to delete the provisioned server.

deviantony

It seems that this version is not working with Edge stacks, I created an Edge stack but it stayed in Pending state for my Edge endpoint (agent running the following implementation feat(edge-jobs): collect logs agent#140)
There is a problem with the schedule date associated to an Edge job, in the following picture I created an Edge job that should start at 4.45 and when I inspect it the associated schedule date is set to 4.42 (current time of my machine)

Rename Tasks to Results

After succesfully downloading the logs, it should automatically clear the logs

chiptus · 2020-06-07T07:19:11Z

t seems that this version is not working with Edge stacks, I created an Edge stack but it stayed in Pending state for my Edge endpoint (agent running the following implementation portainer/agent#140)

which endpoint was it? running on the vagrant setup everything worked as expected. running on a local agent (non swarm), it stayed in pending, but that's expected because the endpoint can't run stacks

chiptus · 2020-06-07T07:29:49Z

After succesfully downloading the logs, it should automatically clear the logs

it seems like FileSaver doesn't return a promise, so there's no way to understand if the file was downloaded

ncresswell · 2020-06-07T07:45:39Z

We need to make sure that non-swarm edge endpoints are not able to be added to an edge group. Rgds, Neil Cresswell On 7/06/2020, at 7:19 PM, Chaim Lev-Ari <[email protected]> wrote: * t seems that this version is not working with Edge stacks, I created an Edge stack but it stayed in Pending state for my Edge endpoint (agent running the following implementation portainer/agent#140<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_agent_pull_140&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=kBRAF30_gGdbpB8kgA86LeEJRICpczXtVzUE_c0np18&s=i1yVANOku4X1j7DmB523MZlTvqQiPV5Cwiz1TXdaJgE&e=>) which endpoint was it? running on the vagrant setup everything worked as expected. running on a local agent (non swarm), it stayed in pending, but that's expected because the endpoint can't run stacks — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_pull_3840-23issuecomment-2D640169689&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=kBRAF30_gGdbpB8kgA86LeEJRICpczXtVzUE_c0np18&s=QXHU-i-C3JoEljuPXXc3HaLf5YONpJdOUZlkT14u5x4&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFQ2XFJJHS7EV4C366UPMXDRVM5Q3ANCNFSM4NGECB3Q&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=kBRAF30_gGdbpB8kgA86LeEJRICpczXtVzUE_c0np18&s=9AXbbbMhrdREBUTWGNhCwjHUCn4zh5AZbLFGC8LFiXg&e=>.

chiptus · 2020-06-07T07:58:21Z

There is a problem with the schedule date associated to an Edge job, in the following picture I created an Edge job that should start at 4.45 and when I inspect it the associated schedule date is set to 4.42 (current time of my machine)

what do you mean inspect it? I got the corn job set at the same time I set in the form. The problem I saw is that when submitting the cron expression I ss prefixed with "0 " thus making it not valid in Linux (which expects 5 places, not 6, as far as I understand). fixing it made the job run in the time I assigned it

deviantony · 2020-06-09T03:45:17Z

Been testing this again.

There is a problem with the schedule date associated to an Edge job, in the following picture I created an Edge job that should start at 4.45 and when I inspect it the associated schedule date is set to 4.42 (current time of my machine)

Cannot reproduce this one.

It seems that this version is not working with Edge stacks, I created an Edge stack but it stayed in Pending state for my Edge endpoint (agent running the following implementation portainer/agent#140)

Cannot reproduce this one.

After succesfully downloading the logs, it should automatically clear the logs

Ignore this for now.

deviantony

@ncresswell can you validate host jobs migration? Migrating from an instance with host jobs defined to this new version.

@chiptus have a look at my comments, have some questions as well as some suggestions.

api/bolt/migrator/migrator.go

api/http/handler/edgejobs/edgejob_inspect.go

api/http/handler/endpoints/endpoint_status_inspect.go

api/http/handler/edgejobtasks/handler.go

api/http/handler/handler.go

api/internal/snapshot/snapshot.go

api/portainer.go

ncresswell · 2020-06-22T21:47:04Z

I cant even use the new image on ANY existing instance.. .get invalid time or something like that. I can only deploy new.

deviantony · 2020-06-22T21:55:13Z

@chiptus this probably requires a fix

deviantony · 2020-06-24T21:34:40Z

@pull-dog down

deviantony

One minor change required, LGTM

api/bolt/migrator/migrator.go

deviantony

Took care of removing the unused dependency, LGTM

deviantony · 2020-06-25T03:25:44Z

Build is OK, CI is failing cause free plan !

fix(edgeconfigs): add triggers for endpoint groups EE-5417 fix(edgeconfigs): fix wrong progress counts EE-4517 fix(edgeconfigs): push missing delete command for async EE-5417

chiptus added the work-in-progress Indicates that the PR is a work-in-progress and not ready yet label May 20, 2020

chiptus requested a review from deviantony May 20, 2020 16:55

chiptus changed the base branch from develop to 2.0 May 20, 2020 16:55

deviantony requested changes May 21, 2020

View reviewed changes

chiptus force-pushed the feat3745-move-host-jobs-to-edge branch from c524a07 to e2afaad Compare May 21, 2020 08:31

chiptus commented May 21, 2020

View reviewed changes

app/edge/components/group-form/groupFormController.js Outdated Show resolved Hide resolved

app/edge/services/edge-job.js Outdated Show resolved Hide resolved

chiptus requested a review from deviantony May 24, 2020 05:38

chiptus removed the work-in-progress Indicates that the PR is a work-in-progress and not ready yet label May 24, 2020

deviantony requested changes May 26, 2020

View reviewed changes

chiptus force-pushed the feat3745-move-host-jobs-to-edge branch from 64fb8ee to c4fe751 Compare May 27, 2020 12:22

chiptus force-pushed the feat3745-move-host-jobs-to-edge branch from cf45858 to 5656986 Compare June 1, 2020 09:48

chiptus requested a review from deviantony June 2, 2020 14:32

chiptus mentioned this pull request Jun 2, 2020

feat(edge-jobs): collect logs portainer/agent#140

Merged

deviantony force-pushed the 2.0 branch from 5b1e6a4 to 334c015 Compare June 2, 2020 23:40

deviantony changed the base branch from 2.0 to develop June 2, 2020 23:42

deviantony added the rebase-required Indicates that the PR must be rebased on the latest development branch label Jun 2, 2020

chiptus force-pushed the feat3745-move-host-jobs-to-edge branch from c6780c2 to 55ef5d5 Compare June 3, 2020 05:07

chiptus force-pushed the feat3745-move-host-jobs-to-edge branch from 55ef5d5 to 302c38b Compare June 4, 2020 06:41

deviantony removed the rebase-required Indicates that the PR must be rebased on the latest development branch label Jun 4, 2020

chiptus force-pushed the feat3745-move-host-jobs-to-edge branch from c2d319b to a946a21 Compare June 4, 2020 11:25

deviantony requested changes Jun 5, 2020

View reviewed changes

chiptus added 6 commits June 22, 2020 10:40

feat(edge-job): set default time for 1 hour from now

d17ef5d

feat(edge-job): add validation for cron format

bd38843

feat(edge-job): add a note about timezone

5919986

fix(edge-job): replace regex

6f99539

fix(edge-job): check for every minute cron

110f410

style(edge-jobs): add reference for cron regex

24d122d

chiptus force-pushed the feat3745-move-host-jobs-to-edge branch from 0ce78b8 to 24d122d Compare June 22, 2020 07:40

deviantony reviewed Jun 22, 2020

View reviewed changes

chiptus added 11 commits June 23, 2020 08:26

refactor(edge-jobs): rename migration name

f5ff935

refactor(edge-job): rename edge job response

1c5c758

refactor(snapshot): rename snapshot endpoint method

0381508

refactor(edge-jobs): move tasks handler to edgejobs

509ecda

feat(security): introduce a middleware for edge compute operations

bdebb15

feat(edge-job): use edge compute middleware

07c6fbe

feat(edge-groups): filter http actions based on edge setting

eb87650

fix(security): return from edge bouncer if failed

f3f4848

feat(edge-stacks): filter http actions based on edge setting

19afa21

feat(edge-groups): show error when failed to load groups

37c1af7

refactor(db): remove edge-jobs migration

30f3aaa

chiptus requested a review from deviantony June 24, 2020 08:58

deviantony requested changes Jun 25, 2020

View reviewed changes

api/bolt/migrator/migrator.go Outdated Show resolved Hide resolved

refactor(migrator): remove unused dependency

f531fc0

deviantony approved these changes Jun 25, 2020

View reviewed changes

deviantony merged commit 24528ec into develop Jun 25, 2020

deviantony deleted the feat3745-move-host-jobs-to-edge branch June 25, 2020 03:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(edge-compute): move host jobs to edge #3840

feat(edge-compute): move host jobs to edge #3840

chiptus commented May 20, 2020

deviantony left a comment •

edited by chiptus

Loading

chiptus left a comment

deviantony left a comment •

edited by chiptus

Loading

chiptus commented May 27, 2020

chiptus commented May 27, 2020 •

edited

Loading

deviantony commented Jun 2, 2020

pull-dog bot commented Jun 3, 2020 •

edited

Loading

deviantony left a comment •

edited

Loading

chiptus commented Jun 7, 2020

chiptus commented Jun 7, 2020

ncresswell commented Jun 7, 2020 via email

chiptus commented Jun 7, 2020

deviantony commented Jun 9, 2020

deviantony left a comment

ncresswell commented Jun 22, 2020 via email

deviantony commented Jun 22, 2020

deviantony commented Jun 24, 2020

deviantony left a comment

deviantony left a comment

deviantony commented Jun 25, 2020

feat(edge-compute): move host jobs to edge #3840

feat(edge-compute): move host jobs to edge #3840

Conversation

chiptus commented May 20, 2020

deviantony left a comment • edited by chiptus Loading

Choose a reason for hiding this comment

chiptus left a comment

Choose a reason for hiding this comment

deviantony left a comment • edited by chiptus Loading

Choose a reason for hiding this comment

chiptus commented May 27, 2020

chiptus commented May 27, 2020 • edited Loading

deviantony commented Jun 2, 2020

pull-dog bot commented Jun 3, 2020 • edited Loading

deviantony left a comment • edited Loading

Choose a reason for hiding this comment

chiptus commented Jun 7, 2020

chiptus commented Jun 7, 2020

ncresswell commented Jun 7, 2020 via email

chiptus commented Jun 7, 2020

deviantony commented Jun 9, 2020

deviantony left a comment

Choose a reason for hiding this comment

ncresswell commented Jun 22, 2020 via email

deviantony commented Jun 22, 2020

deviantony commented Jun 24, 2020

deviantony left a comment

Choose a reason for hiding this comment

deviantony left a comment

Choose a reason for hiding this comment

deviantony commented Jun 25, 2020

deviantony left a comment •

edited by chiptus

Loading

deviantony left a comment •

edited by chiptus

Loading

chiptus commented May 27, 2020 •

edited

Loading

pull-dog bot commented Jun 3, 2020 •

edited

Loading

deviantony left a comment •

edited

Loading