Build damaging/goodfaith models for Romanian Wikipedia
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Halfak
	Jan 27 2017, 5:33 PM

Related Objects
Search...

Status	Assigned	Task
Resolved	awight	T187836 [Epic] Audit of pending ORES GUI deployments
Resolved	awight	T156518 Deploy ORES Review Tool on Romanian Wikipedia
Resolved	• Catrope	T170723 Deploy ORES Review Tool & ORES-based RCFilters for Romanian & Albanian Wikipedia
Resolved	awight	T170485 ORES deployment - Mid July, 2017
Duplicate	None	T156501 Enable ORES Review Tool in Romanian Wikipedia
Resolved	None	T130213 [Epic] Edit quality models (damaging/goodfaith)
Resolved	Halfak	T166045 Scoring platform team FY18 Q1
Resolved	Sumit	T156503 Build damaging/goodfaith models for Romanian Wikipedia
Resolved	Halfak	T156517 Complete Romanian Wikipedia edit quality campaign
Resolved	None	T156357 Deploy edit quality campaign for Romanian Wikipedia

Event Timeline

Halfak created this task.Jan 27 2017, 5:33 PM

Halfak added a subtask: T156357: Deploy edit quality campaign for Romanian Wikipedia.

Halfak added a parent task: T156518: Deploy ORES Review Tool on Romanian Wikipedia.Jan 27 2017, 7:31 PM

Halfak mentioned this in T156357: Deploy edit quality campaign for Romanian Wikipedia.Jan 27 2017, 7:33 PM

Halfak added a subtask: T156517: Complete Romanian Wikipedia edit quality campaign.Feb 2 2017, 3:26 PM

Halfak removed a subtask: T156357: Deploy edit quality campaign for Romanian Wikipedia.

Halfak triaged this task as Low priority.Feb 2 2017, 3:31 PM

Halfak moved this task from Unsorted to New development on the Machine-Learning-Team board.

Halfak added a parent task: T130213: [Epic] Edit quality models (damaging/goodfaith).Apr 14 2017, 4:44 PM

Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.Jun 24 2017, 6:17 PM

Restricted Application added a project: artificial-intelligence. · View Herald TranscriptJun 24 2017, 6:17 PM

Halfak assigned this task to Sumit.Jun 26 2017, 4:23 PM

Damaging:

make models/rowiki.damaging.gradient_boosting.model       
cat datasets/rowiki.labeled_revisions.w_cache.20k_2016.json | \
        revscoring cv_train \
                revscoring.scorer_models.GradientBoosting \
                editquality.feature_lists.rowiki.damaging \
                damaging \
                --version=0.3.0 \
                -p 'max_depth=5' \
                -p 'learning_rate=0.01' \
                -p 'max_features="log2"' \
                -p 'n_estimators=700' \
                -s 'table' -s 'accuracy' -s 'precision' -s 'recall' -s 'pr' -s 'roc' -s 'recall_at_fpr(max_fpr=0.10)' -s 'filter_rate_at_recall(min_recall=0.9)' -s 'filt
er_rate_at_recall(min_recall=0.75)' -s 'recall_at_precision(min_precision=0.995)' -s 'recall_at_precision(min_precision=0.99)' -s 'recall_at_precision(min_precision=0.98
)' -s 'recall_at_precision(min_precision=0.90)' -s 'recall_at_precision(min_precision=0.75)' -s 'recall_at_precision(min_precision=0.60)' -s 'recall_at_precision(min_pre
cision=0.45)' -s 'recall_at_precision(min_precision=0.15)' \
                --balance-sample-weight \
                --center --scale > models/rowiki.damaging.gradient_boosting.model
2017-06-27 08:00:43,699 INFO:revscoring.utilities.cv_train -- Cross-validating model statistics for 10 folds...
2017-06-27 08:00:44,352 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 1...
2017-06-27 08:03:13,756 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 2...
2017-06-27 08:05:56,730 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 3...
2017-06-27 08:08:40,903 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 4...
2017-06-27 08:11:27,209 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 5...
2017-06-27 08:14:17,733 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 6...
2017-06-27 08:17:35,238 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 7...
2017-06-27 08:20:17,584 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 8...
2017-06-27 08:23:33,992 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 9...
2017-06-27 08:26:46,826 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 10...
2017-06-27 08:29:07,741 INFO:revscoring.utilities.cv_train -- Training model on all data...
ScikitLearnClassifier
 - type: GradientBoosting
 - params: max_leaf_nodes=null, learning_rate=0.01, min_samples_split=2, verbose=0, center=true, warm_start=false, n_estimators=700, presort="auto", balanced_sample_weig
ht=true, loss="deviance", min_samples_leaf=1, balanced_sample=false, init=null, random_state=null, subsample=1.0, max_features="log2", scale=true, min_weight_fraction_le
af=0.0, max_depth=5
 - version: 0.3.0
 - trained: 2017-06-27T08:29:32.376824

Table:
                 ~False    ~True
        -----  --------  -------
        False     17212     1678
        True        109      828

Accuracy: 0.91
Precision:
        -----  -----
        False  0.994
        True   0.33
        -----  -----

Recall:
        -----  -----
        False  0.911
        True   0.881
        -----  -----

PR-AUC:
        -----  -----
        False  0.994
        True   0.54
        -----  -----

ROC-AUC:
        -----  -----
        False  0.959
        True   0.963
        -----  -----

Recall @ 0.1 false-positive rate:
        label      threshold    recall    fpr
        -------  -----------  --------  -----
        False          0.548     0.904  0.094
        True           0.444     0.916  0.098

Filter rate @ 0.9 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.569          0.138     0.9
        True           0.455          0.866     0.906

Filter rate @ 0.75 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.915          0.284     0.75
        True           0.777          0.915     0.752

Recall @ 0.995 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.564     0.901        0.995
        True           0.962     0.038        1

Recall @ 0.99 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.331     0.934         0.99
        True           0.962     0.038         1

Recall @ 0.98 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.128     0.968         0.98
        True           0.962     0.038         1

Recall @ 0.9 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.959     0.051        0.981

Recall @ 0.75 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.945     0.164        0.787

Recall @ 0.6 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.923     0.351        0.608

Recall @ 0.45 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.827     0.706        0.454

Recall @ 0.15 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.105     0.981        0.189

Goodfaith:

make models/rowiki.goodfaith.gradient_boosting.model                                     [97/1922]
cat datasets/rowiki.labeled_revisions.w_cache.20k_2016.json | \
        revscoring cv_train \
                revscoring.scorer_models.GradientBoosting \
                editquality.feature_lists.rowiki.goodfaith \
                goodfaith \
                --version=0.3.0 \
                -p 'max_depth=3' \
                -p 'learning_rate=0.1' \
                -p 'max_features="log2"' \
                -p 'n_estimators=300' \
                -s 'table' -s 'accuracy' -s 'precision' -s 'recall' -s 'pr' -s 'roc' -s 'recall_at_fpr(max_fpr=0.10)' -s 'filter_rate_at_recall(min_recall=0.9)' -s 'filt
er_rate_at_recall(min_recall=0.75)' -s 'recall_at_precision(min_precision=0.995)' -s 'recall_at_precision(min_precision=0.99)' -s 'recall_at_precision(min_precision=0.98
)' -s 'recall_at_precision(min_precision=0.90)' -s 'recall_at_precision(min_precision=0.75)' -s 'recall_at_precision(min_precision=0.60)' -s 'recall_at_precision(min_pre
cision=0.45)' -s 'recall_at_precision(min_precision=0.15)' \
                --balance-sample-weight \
                --center --scale > models/rowiki.goodfaith.gradient_boosting.model
2017-06-27 13:11:03,053 INFO:revscoring.utilities.cv_train -- Cross-validating model statistics for 10 folds...
2017-06-27 13:11:03,907 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 1...
2017-06-27 13:13:54,482 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 2...
2017-06-27 13:17:12,485 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 3...
2017-06-27 13:19:46,401 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 4...
2017-06-27 13:22:17,370 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 5...
2017-06-27 13:25:08,119 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 6...
2017-06-27 13:27:31,615 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 7...
2017-06-27 13:29:51,620 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 8...
2017-06-27 13:32:09,126 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 9...
2017-06-27 13:34:20,776 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 10...
2017-06-27 13:36:25,349 INFO:revscoring.utilities.cv_train -- Training model on all data...
ScikitLearnClassifier
 - type: GradientBoosting
 - params: max_features="log2", min_samples_leaf=1, min_weight_fraction_leaf=0.0, warm_start=false, balanced_sample=false, balanced_sample_weight=true, center=true, loss
="deviance", min_samples_split=2, max_leaf_nodes=null, verbose=0, max_depth=3, random_state=null, n_estimators=300, learning_rate=0.1, scale=true, subsample=1.0, init=nu
ll, presort="auto"
 - version: 0.3.0
 - trained: 2017-06-27T13:36:32.777290

Table:
                 ~False    ~True
        -----  --------  -------
        False       489       81
        True       1393    17864

Accuracy: 0.926
Precision:
        -----  -----
        False  0.26
        True   0.995
        -----  -----

Recall:
        -----  -----
        False  0.856
        True   0.928
        -----  -----

PR-AUC:
        -----  -----
        False  0.469
        True   0.994
        -----  -----

ROC-AUC:
        -----  -----
        False  0.964
        True   0.962
        -----  -----

Recall @ 0.1 false-positive rate:
        label      threshold    recall    fpr
        -------  -----------  --------  -----
        False          0.278     0.93   0.097
        True           0.634     0.914  0.093

Filter rate @ 0.9 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.37           0.89      0.907
        True           0.738          0.124     0.9

Filter rate @ 0.75 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.76           0.933     0.754
        True           0.972          0.271     0.75

Recall @ 0.995 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04         1
        True           0.414     0.936        0.995

Recall @ 0.99 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04         1
        True           0.174     0.964        0.991

Recall @ 0.98 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04          1
        True           0.059     0.991         0.98

Recall @ 0.9 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.986     0.052        0.99
        True           0.013     1            0.972

Recall @ 0.75 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.977     0.107        0.808
        True           0.013     1            0.972

Recall @ 0.6 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.951     0.296        0.616
        True           0.013     1            0.972

Recall @ 0.45 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.914      0.49        0.46
        True           0.013      1           0.972

Recall @ 0.15 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.098     0.968        0.161
        True           0.013     1            0.972

need to retrain the models after the regex update, PR soon.

https://github.com/wiki-ai/editquality/pull/78

Sumit moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Jun 27 2017, 7:01 PM

Halfak added a parent task: T166045: Scoring platform team FY18 Q1.Jun 28 2017, 5:03 PM

Sumit moved this task from Review to Completed on the Machine-Learning-Team (Active Tasks) board.Jul 7 2017, 7:27 PM

Halfak added a parent task: T170485: ORES deployment - Mid July, 2017.Jul 12 2017, 9:30 PM

Halfak closed this task as Resolved.Jul 12 2017, 10:49 PM

Halfak closed subtask T156517: Complete Romanian Wikipedia edit quality campaign as Resolved.