Skip to content

Commit

Permalink
Merge pull request tensorflow#1771 from 8bitmp3:lint-estimator-guide
Browse files Browse the repository at this point in the history 
PiperOrigin-RevId: 348090159
  • Loading branch information
@copybara-github
copybara-github committed Dec 17, 2020
2 parents 5ba6e25 e9f1ce0 commit 200cdb8
Showing 1 changed file with 32 additions and 42 deletions.
74 changes: 32 additions & 42 deletions site/en/guide/estimator.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -71,14 71,14 @@
"This document introduces `tf.estimator`—a high-level TensorFlow\n",
"API. Estimators encapsulate the following actions:\n",
"\n",
"* training\n",
"* evaluation\n",
"* prediction\n",
"* export for serving\n",
"* Training\n",
"* Evaluation\n",
"* Prediction\n",
"* Export for serving\n",
"\n",
"TensorFlow implements several pre-made Estimators. Custom estimators are still suported, but mainly as a backwards compatibility measure. **Custom estimators should not be used for new code**. All Estimators--whether pre-made or custom--are classes based on the `tf.estimator.Estimator` class.\n",
"TensorFlow implements several pre-made Estimators. Custom estimators are still suported, but mainly as a backwards compatibility measure. **Custom estimators should not be used for new code**. All Estimators—pre-made or custom ones—are classes based on the `tf.estimator.Estimator` class.\n",
"\n",
"For a quick example try [Estimator tutorials](../tutorials/estimator/linear.ipynb). For an overview of the API design, see the [white paper](https://arxiv.org/abs/1708.02637)."
"For a quick example, try [Estimator tutorials](../tutorials/estimator/linear.ipynb). For an overview of the API design, check the [white paper](https://arxiv.org/abs/1708.02637)."
]
},
{
Expand All @@ -98,7 98,7 @@
},
"outputs": [],
"source": [
"! pip install -U tensorflow_datasets"
"!pip install -U tensorflow_datasets"
]
},
{
Expand Down Expand Up @@ -127,7 127,7 @@
"Similar to a `tf.keras.Model`, an `estimator` is a model-level abstraction. The `tf.estimator` provides some capabilities currently still under development for `tf.keras`. These are:\n",
"\n",
" * Parameter server based training\n",
" * Full [TFX](http://tensorflow.org/tfx) integration."
" * Full [TFX](http://tensorflow.org/tfx) integration"
]
},
{
Expand All @@ -141,14 141,12 @@
"\n",
"* You can run Estimator-based models on a local host or on a distributed multi-server environment without changing your model. Furthermore, you can run Estimator-based models on CPUs, GPUs, or TPUs without recoding your model.\n",
"* Estimators provide a safe distributed training loop that controls how and when to: \n",
" * load data\n",
" * handle exceptions\n",
" * create checkpoint files and recover from failures\n",
" * save summaries for TensorBoard\n",
" * Load data\n",
" * Handle exceptions\n",
" * Create checkpoint files and recover from failures\n",
" * Save summaries for TensorBoard\n",
"\n",
"When writing an application with Estimators, you must separate the data input\n",
"pipeline from the model. This separation simplifies experiments with\n",
"different data sets."
"When writing an application with Estimators, you must separate the data input pipeline from the model. This separation simplifies experiments with different datasets."
]
},
{
Expand Down Expand Up @@ -226,7 224,7 @@
"- The second uses the `class` feature as a categorical input.\n",
"- The third uses the `embark_town` as a categorical input, but uses the `hashing trick` to avoid the need to enumerate the options, and to set the number of options.\n",
"\n",
"For further information, see the [feature columns tutorial](https://www.tensorflow.org/tutorials/keras/feature_columns)."
"For further information, check the [feature columns tutorial](https://www.tensorflow.org/tutorials/keras/feature_columns)."
]
},
{
Expand Down Expand Up @@ -275,7 273,7 @@
"id": "QGl9oYuFoYj6"
},
"source": [
"For further information, see the [linear classifier tutorial](https://www.tensorflow.org/tutorials/estimator/linear)."
"For more information, you can go the [linear classifier tutorial](https://www.tensorflow.org/tutorials/estimator/linear)."
]
},
{
Expand Down Expand Up @@ -372,16 370,6 @@
"Instantiate a Keras MobileNet V2 model and compile the model with the optimizer, loss, and metrics to train with:"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "D4ro6RAj2UoW"
},
"source": [
"import tensorflow as tf\n",
"import tensorflow_datasets as tfds"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand Down Expand Up @@ -637,7 625,7 @@
"source": [
"## SavedModels from Estimators\n",
"\n",
"Estimators export SavedModels through [`tf.Estimator.export_saved_model`](https://www.tensorflow.org/api_docs/python/tf/estimator/Estimator#export_saved_model)."
"Estimators export SavedModels through `tf.Estimator.export_saved_model`."
]
},
{
Expand Down Expand Up @@ -750,11 738,11 @@
"source": [
"## Using `tf.distribute.Strategy` with Estimator (Limited support)\n",
"\n",
"See the [Distributed training guide](guide/distributed_training.ipynb) for more info.\n",
"`tf.estimator` is a distributed training TensorFlow API that originally supported the async parameter server approach. `tf.estimator` now supports `tf.distribute.Strategy`. If you're using `tf.estimator`, you can change to distributed training with very few changes to your code. With this, Estimator users can now do synchronous distributed training on multiple GPUs and multiple workers, as well as use TPUs. This support in Estimator is, however, limited. Check out the [What's supported now](#estimator_support) section below for more details.\n",
"\n",
"`tf.estimator` is a distributed training TensorFlow API that originally supported the async parameter server approach. `tf.estimator` now supports `tf.distribute.Strategy`. If you're using `tf.estimator`, you can change to distributed training with very few changes to your code. With this, Estimator users can now do synchronous distributed training on multiple GPUs and multiple workers, as well as use TPUs. This support in Estimator is, however, limited. See [What's supported now](#estimator_support) section below for more details.\n",
"Using `tf.distribute.Strategy` with Estimator is slightly different than in the Keras case. Instead of using `strategy.scope`, now you pass the strategy object into the `RunConfig` for the Estimator.\n",
"\n",
"The usage of `tf.distribute.Strategy` with Estimator is slightly different than the Keras case. Instead of using `strategy.scope`, now we pass the strategy object into the [`RunConfig`](https://www.tensorflow.org/api_docs/python/tf/estimator/RunConfig) for the Estimator.\n",
"You can refer to the [distributed training guide](distributed_training.ipynb) for more information.\n",
"\n",
"Here is a snippet of code that shows this with a premade Estimator `LinearRegressor` and `MirroredStrategy`:\n"
]
Expand Down Expand Up @@ -782,9 770,9 @@
"id": "n6eSfLN5RGY8"
},
"source": [
"We use a premade Estimator here, but the same code works with a custom Estimator as well. `train_distribute` determines how training will be distributed, and `eval_distribute` determines how evaluation will be distributed. This is another difference from Keras where we use the same strategy for both training and eval.\n",
"Here, you use a premade Estimator, but the same code works with a custom Estimator as well. `train_distribute` determines how training will be distributed, and `eval_distribute` determines how evaluation will be distributed. This is another difference from Keras where you use the same strategy for both training and eval.\n",
"\n",
"Now we can train and evaluate this Estimator with an input function:\n"
"Now you can train and evaluate this Estimator with an input function:\n"
]
},
{
Expand All @@ -808,11 796,11 @@
"id": "hgaU9xQSSk2x"
},
"source": [
"Another difference to highlight here between Estimator and Keras is the input handling. In Keras, we mentioned that each batch of the dataset is split automatically across the multiple replicas. In Estimator, however, we do not do automatic splitting of batch, nor automatically shard the data across different workers. You have full control over how you want your data to be distributed across workers and devices, and you must provide an `input_fn` to specify how to distribute your data.\n",
"Another difference to highlight here between Estimator and Keras is the input handling. In Keras, each batch of the dataset is split automatically across the multiple replicas. In Estimator, however, you do not perform automatic batch splitting, nor automatically shard the data across different workers. You have full control over how you want your data to be distributed across workers and devices, and you must provide an `input_fn` to specify how to distribute your data.\n",
"\n",
"Your `input_fn` is called once per worker, thus giving one dataset per worker. Then one batch from that dataset is fed to one replica on that worker, thereby consuming N batches for N replicas on 1 worker. In other words, the dataset returned by the `input_fn` should provide batches of size `PER_REPLICA_BATCH_SIZE`. And the global batch size for a step can be obtained as `PER_REPLICA_BATCH_SIZE * strategy.num_replicas_in_sync`.\n",
"\n",
"When doing multi worker training, you should either split your data across the workers, or shuffle with a random seed on each. You can see an example of how to do this in the [Multi-worker Training with Estimator](../tutorials/distribute/multi_worker_with_estimator.ipynb)."
"When performing multi-worker training, you should either split your data across the workers, or shuffle with a random seed on each. You can check an example of how to do this in the [Multi-worker training with Estimator](../tutorials/distribute/multi_worker_with_estimator.ipynb) tutorial."
]
},
{
Expand All @@ -831,20 819,22 @@
},
"source": [
"<a name=\"estimator_support\"></a>\n",
"\n",
"### What's supported now?\n",
"\n",
"There is limited support for training with Estimator using all strategies except `TPUStrategy`. Basic training and evaluation should work, but a number of advanced features such as `v1.train.Scaffold` do not. There may also be a number of bugs in this integration. At this time, we do not plan to actively improve this support, and instead are focused on Keras and custom training loop support. If at all possible, you should prefer to use `tf.distribute` with those APIs instead.\n",
"There is limited support for training with Estimator using all strategies except `TPUStrategy`. Basic training and evaluation should work, but a number of advanced features such as `v1.train.Scaffold` do not. There may also be a number of bugs in this integration and there are no plans to actively improve this support (the focus is on Keras and custom training loop support). If at all possible, you should prefer to use `tf.distribute` with those APIs instead.\n",
"\n",
"| Training API \t| MirroredStrategy \t| TPUStrategy \t| MultiWorkerMirroredStrategy \t| CentralStorageStrategy \t| ParameterServerStrategy \t|\n",
"|:---------------\t|:------------------\t|:-------------\t|:-----------------------------\t|:------------------------\t|:-------------------------\t|\n",
"| Estimator API \t| Limited Support \t| Not supported \t| Limited Support \t| Limited Support \t| Limited Support \t|\n",
"| Estimator API \t| Limited support \t| Not supported \t| Limited support \t| Limited support \t| Limited support \t|\n",
"\n",
"### Examples and tutorials\n",
"\n",
"### Examples and Tutorials\n",
"Here are some examples that show end to end usage of various strategies with Estimator:\n",
"Here are some end-to-end examples that show how to use various strategies with Estimator:\n",
"\n",
"1. [Multi-worker Training with Estimator](../tutorials/distribute/multi_worker_with_estimator.ipynb) to train MNIST with multiple workers using `MultiWorkerMirroredStrategy`.\n",
"2. [End to end example](https://github.com/tensorflow/ecosystem/tree/master/distribution_strategy) for multi worker training in tensorflow/ecosystem using Kubernetes templates. This example starts with a Keras model and converts it to an Estimator using the `tf.keras.estimator.model_to_estimator` API.\n",
"3. Official [ResNet50](https://github.com/tensorflow/models/blob/master/official/vision/image_classification/resnet_imagenet_main.py) model, which can be trained using either `MirroredStrategy` or `MultiWorkerMirroredStrategy`."
"1. The [Multi-worker Training with Estimator tutorial](../tutorials/distribute/multi_worker_with_estimator.ipynb) shows how you can train with multiple workers using `MultiWorkerMirroredStrategy` on the MNIST dataset.\n",
"2. An end-to-end example of [running multi-worker training with distribution strategies](https://github.com/tensorflow/ecosystem/tree/master/distribution_strategy) in `tensorflow/ecosystem` using Kubernetes templates. It starts with a Keras model and converts it to an Estimator using the `tf.keras.estimator.model_to_estimator` API.\n",
"3. The official [ResNet50](https://github.com/tensorflow/models/blob/master/official/vision/image_classification/resnet_imagenet_main.py) model, which can be trained using either `MirroredStrategy` or `MultiWorkerMirroredStrategy`."
]
}
],
Expand Down

0 comments on commit 200cdb8

Please sign in to comment.