三人行,必有我师焉。择其善者而从之,其不善者而改之。——孔子
/** * mapPartitions(func) 用法: rdd2=rdd1.mapPartitions(func) 语义: 对rdd1中每个分区进行func操作,输出新的一组数据 * */ public static void mapPartitions() { //创建SparkConf SparkConf conf = new SparkConf().setAppName("map").setMaster("local"); //创建javaSparkContext JavaSparkContext sc = new JavaSparkContext(conf); JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4)); JavaRDD<Tuple2<Integer, Integer>> tuple2JavaRDD = rdd.mapPartitions(new FlatMapFunction<Iterator<Integer>, Tuple2<Integer, Integer>>() { @Override public Iterator<Tuple2<Integer, Integer>> call(Iterator<Integer> it) throws Exception { ArrayList<Tuple2<Integer, Integer>> tuple2s = new ArrayList<>(); while (it.hasNext()) { Integer newValue=0; Integer value=it.next(); if(value%2==0) newValue=value*value; else newValue=value; tuple2s.add(new Tuple2<>(value,newValue)); } return tuple2s.iterator(); } }); tuple2JavaRDD.foreach(new VoidFunction<Tuple2<Integer, Integer>>() { @Override public void call(Tuple2<Integer, Integer> integerIntegerTuple2) throws Exception { System.out.println(integerIntegerTuple2); } }); }
答案: B
;
版权声明:
本文为智客工坊「楠木大叔」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。