matlab.compiler.mlspark.RDD Class
R2026bNamespace: matlab.compiler.mlspark
Superclasses:
(To be removed) Interface class to represent a Spark Resilient Distributed Dataset (RDD)
matlab.compiler.mlspark.RDD will be removed in a future
release. There is no replacement functionality. For more information, see Version History.
Description
A Resilient Distributed Dataset or RDD is a programming abstraction in Spark™. It represents a collection of elements distributed across many nodes that can be operated in parallel. All work in Spark is expressed as either creating new RDDs, transforming existing RDDs, or calling operations on RDDs to compute a result. You can create RDDs in two ways:
By loading an external dataset
By parallelizing a collection of objects in the driver program
Once created, two types of operations can be performed using RDDs: transformations and actions.
Construction
An RDD object can only be created using the
methods of the SparkContext class. A collection of SparkContext methods
used to create RDDs is listed below for convenience. See the documentation
of the SparkContext class
for more information.
| SparkContext Method Name | Purpose |
|---|---|
parallelize | Create an RDD from local MATLAB® values |
datastoreToRDD | Convert MATLAB |
textFile | Create an RDD from a text file |
Once an RDD has been created using a method from the SparkContext class,
you can use any of the methods in the RDD class to
manipulate your RDD.
Properties
The properties of this class are hidden.
Methods
Transformations
| aggregateByKey | (To be removed) Aggregate the values of each key, using given combine functions and a neutral “zero value” |
| cartesian | (To be removed) Create an RDD that is the Cartesian product of two RDDs |
| coalesce | (To be removed) Reduce the number of partitions in an RDD |
| cogroup | (To be removed) Group data from RDDs sharing the same key |
| combineByKey | (To be removed) Combine the elements for each key using a custom set of aggregation functions |
| distinct | (To be removed) Return a new RDD containing the distinct elements of an existing RDD |
| filter | (To be removed) Return a new RDD containing only the elements that satisfy a predicate function |
| flatMap | (To be removed) Return a new RDD by first applying a function to all elements of an existing RDD, and then flattening the results |
| flatMapValues | (To be removed) Pass each value in the key-value pair RDD through a flatMap method
without changing the keys |
| foldByKey | (To be removed) Merge the values for each key using an associative function and a neutral “zero value” |
| fullOuterJoin | (To be removed) Perform a full outer join between two key-value pair RDDs |
| glom | (To be removed) Coalesce all elements within each partition of an RDD |
| groupBy | (To be removed) Return an RDD of grouped items |
| groupByKey | (To be removed) Group the values for each key in the RDD into a single sequence |
| intersection | (To be removed) Return the set intersection of one RDD with another |
| join | (To be removed) Return an RDD containing all pairs of elements with matching keys |
| keyBy | (To be removed) Create tuples of the elements in an RDD by applying a function |
| keys | (To be removed) Return an RDD with the keys of each tuple |
| leftOuterJoin | (To be removed) Perform a left outer join |
| map | (To be removed) Return a new RDD by applying a function to each element of an input RDD |
| mapValues | (To be removed) Pass each value in a key-value pair RDD through a map function without modifying the keys |
| reduceByKey | (To be removed) Merge the values for each key using an associative reduce function |
| repartition | (To be removed) Return a new RDD that has exactly numPartitions partitions |
| rightOuterJoin | (To be removed) Perform a right outer join |
| sortBy | (To be removed) Sort an RDD by a given function |
| sortByKey | (To be removed) Sort RDD consisting of key-value pairs by key |
| subtract | (To be removed) Return the values resulting from the set difference between two RDDs |
| subtractByKey | (To be removed) Return key-value pairs resulting from the set difference of keys between two RDDs |
| union | (To be removed) Return the set union of one RDD with another |
| values | (To be removed) Return an RDD with the values of each tuple |
| zip | (To be removed) Zip one RDD with another |
| zipWithIndex | (To be removed) Zip an RDD with its element indices |
| zipWithUniqueId | (To be removed) Zip an RDD with generated unique Long IDs |
Actions
| aggregate | (To be removed) Aggregate the elements of each partition and subsequently the results for all partitions into a single value |
| collect | (To be removed) Return a MATLAB cell array that contains all of the elements in an RDD |
| collectAsMap | (To be removed) Return the key-value pairs in an RDD as a MATLAB containers.Map object |
| count | (To be removed) Count number of elements in an RDD |
| fold | (To be removed) Aggregate elements of each partition and the subsequent results for all partitions |
| reduce | (To be removed) Reduce elements of an RDD using the specified commutative and associative function |
| reduceByKeyLocally | (To be removed) Merge the values for each key using an associative reduce function, but return the results immediately to the driver |
| saveAsKeyValueDatastore | (To be removed) Save key-value RDD as a binary file that can be read back
using the datastore function |
| saveAsTallDatastore | (To be removed) Save RDD as a MATLAB tall array to a binary file
that can be read back using the datastore function |
| saveAsTextFile | (To be removed) Save RDD as a text file |
Operations
| cache | (To be removed) Store an RDD in memory |
| checkpoint | (To be removed) Mark an RDD for checkpointing |
| getCheckpointFile | (To be removed) Get the name of the file to which an RDD is checkpointed |
| getDefaultReducePartitions | (To be removed) Get the number of default reduce partitions in an RDD |
| getNumPartitions | (To be removed) Return the number of partitions in an RDD |
| isEmpty | (To be removed) Determine if an RDD contains any elements |
| keyLimit | (To be removed) Return threshold of unique keys that can be stored before spilling to disk |
| persist | (To be removed) Set the value of an RDD’s storage level to persist across operations after it is computed |
| toDebugString | (To be removed) Obtain a description of an RDD and its recursive dependencies for debugging |
| unpersist | (To be removed) Mark an RDD as nonpersistent, remove all blocks for it from memory and disk |
More About
References
See the latest Spark documentation for more information.
