google_api_dataflow v0.5.0 GoogleApi.Dataflow.V1b3.Model.WorkerPool View Source
Describes one particular pool of Cloud Dataflow workers to be instantiated by the Cloud Dataflow service in order to perform the computations required by a job. Note that a workflow job may use multiple pools, in order to match the various computational requirements of the various stages of the job.
Attributes
- autoscalingSettings (AutoscalingSettings): Settings for autoscaling of this WorkerPool. Defaults to:
null
. - dataDisks ([Disk]): Data disks that are used by a VM in this workflow. Defaults to:
null
. defaultPackageSet (String.t): The default package set to install. This allows the service to select a default set of packages which are useful to worker harnesses written in a particular language. Defaults to:
null
.- Enum - one of [DEFAULT_PACKAGE_SET_UNKNOWN, DEFAULT_PACKAGE_SET_NONE, DEFAULT_PACKAGE_SET_JAVA, DEFAULT_PACKAGE_SET_PYTHON]
- diskSizeGb (integer()): Size of root disk for VMs, in GB. If zero or unspecified, the service will attempt to choose a reasonable default. Defaults to:
null
. - diskSourceImage (String.t): Fully qualified source image for disks. Defaults to:
null
. - diskType (String.t): Type of root disk for VMs. If empty or unspecified, the service will attempt to choose a reasonable default. Defaults to:
null
. ipConfiguration (String.t): Configuration for VM IPs. Defaults to:
null
.- Enum - one of [WORKER_IP_UNSPECIFIED, WORKER_IP_PUBLIC, WORKER_IP_PRIVATE]
- kind (String.t): The kind of the worker pool; currently only `harness` and `shuffle` are supported. Defaults to:
null
. - machineType (String.t): Machine type (e.g. "n1-standard-1"). If empty or unspecified, the service will attempt to choose a reasonable default. Defaults to:
null
. - metadata (%{optional(String.t) => String.t}): Metadata to set on the Google Compute Engine VMs. Defaults to:
null
. - network (String.t): Network to which VMs will be assigned. If empty or unspecified, the service will use the network "default". Defaults to:
null
. - numThreadsPerWorker (integer()): The number of threads per worker harness. If empty or unspecified, the service will choose a number of threads (according to the number of cores on the selected machine type for batch, or 1 by convention for streaming). Defaults to:
null
. - numWorkers (integer()): Number of Google Compute Engine workers in this pool needed to execute the job. If zero or unspecified, the service will attempt to choose a reasonable default. Defaults to:
null
. - onHostMaintenance (String.t): The action to take on host maintenance, as defined by the Google Compute Engine API. Defaults to:
null
. - packages ([Package]): Packages to be installed on workers. Defaults to:
null
. - poolArgs (%{optional(String.t) => String.t}): Extra arguments for this worker pool. Defaults to:
null
. - subnetwork (String.t): Subnetwork to which VMs will be assigned, if desired. Expected to be of the form "regions/REGION/subnetworks/SUBNETWORK". Defaults to:
null
. - taskrunnerSettings (TaskRunnerSettings): Settings passed through to Google Compute Engine workers when using the standard Dataflow task runner. Users should ignore this field. Defaults to:
null
. teardownPolicy (String.t): Sets the policy for determining when to turndown worker pool. Allowed values are: `TEARDOWN_ALWAYS`, `TEARDOWN_ON_SUCCESS`, and `TEARDOWN_NEVER`. `TEARDOWN_ALWAYS` means workers are always torn down regardless of whether the job succeeds. `TEARDOWN_ON_SUCCESS` means workers are torn down if the job succeeds. `TEARDOWN_NEVER` means the workers are never torn down. If the workers are not torn down by the service, they will continue to run and use Google Compute Engine VM resources in the user's project until they are explicitly terminated by the user. Because of this, Google recommends using the `TEARDOWN_ALWAYS` policy except for small, manually supervised test jobs. If unknown or unspecified, the service will attempt to choose a reasonable default. Defaults to:
null
.- Enum - one of [TEARDOWN_POLICY_UNKNOWN, TEARDOWN_ALWAYS, TEARDOWN_ON_SUCCESS, TEARDOWN_NEVER]
- workerHarnessContainerImage (String.t): Required. Docker container image that executes the Cloud Dataflow worker harness, residing in Google Container Registry. Defaults to:
null
. - zone (String.t): Zone to run the worker pools in. If empty or unspecified, the service will attempt to choose a reasonable default. Defaults to:
null
.
Link to this section Summary
Functions
Unwrap a decoded JSON object into its complex fields.
Link to this section Types
t()
View Source
t() :: %GoogleApi.Dataflow.V1b3.Model.WorkerPool{
autoscalingSettings: GoogleApi.Dataflow.V1b3.Model.AutoscalingSettings.t(),
dataDisks: [GoogleApi.Dataflow.V1b3.Model.Disk.t()],
defaultPackageSet: any(),
diskSizeGb: any(),
diskSourceImage: any(),
diskType: any(),
ipConfiguration: any(),
kind: any(),
machineType: any(),
metadata: map(),
network: any(),
numThreadsPerWorker: any(),
numWorkers: any(),
onHostMaintenance: any(),
packages: [GoogleApi.Dataflow.V1b3.Model.Package.t()],
poolArgs: map(),
subnetwork: any(),
taskrunnerSettings: GoogleApi.Dataflow.V1b3.Model.TaskRunnerSettings.t(),
teardownPolicy: any(),
workerHarnessContainerImage: any(),
zone: any()
}
t() :: %GoogleApi.Dataflow.V1b3.Model.WorkerPool{ autoscalingSettings: GoogleApi.Dataflow.V1b3.Model.AutoscalingSettings.t(), dataDisks: [GoogleApi.Dataflow.V1b3.Model.Disk.t()], defaultPackageSet: any(), diskSizeGb: any(), diskSourceImage: any(), diskType: any(), ipConfiguration: any(), kind: any(), machineType: any(), metadata: map(), network: any(), numThreadsPerWorker: any(), numWorkers: any(), onHostMaintenance: any(), packages: [GoogleApi.Dataflow.V1b3.Model.Package.t()], poolArgs: map(), subnetwork: any(), taskrunnerSettings: GoogleApi.Dataflow.V1b3.Model.TaskRunnerSettings.t(), teardownPolicy: any(), workerHarnessContainerImage: any(), zone: any() }
Link to this section Functions
decode(value, options) View Source
Unwrap a decoded JSON object into its complex fields.