How to iterate though each dictionary from the list in Scala?-CodePudding

I have a JSON string, which basically is a list of dictionaries. I am trying to use a for loop to iterate through it. Basically in each iteration, I should be able to get the keys such as index, name, source, s3. Can someone please help? So in first iteration I should be able to get index=1,source=a etc.

import scala.util.parsing.json._
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }

object M extends CC[Map[String, Any]]
object A extends CC[List[Any]] //for s3
object I extends CC[Double]
object S extends CC[String]
object E extends CC[String]
object F extends CC[String]
object G extends CC[Map[String, Any]]

val jsonString =
    """
    [{
        "index": 1,
      "source": "a",
      "name": "v",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     },
     {
        "index": 2,
      "source": "b",
      "name": "b",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     }]
     
    """.stripMargin
    
println(List(JSON.parseFull(jsonString)) )

CodePudding user response：

In your example you are trying to use extractors. The extractors based on unapply method. Usually it takes objet and return it arguments. To implement this, you need to define extractors for each type that you want to use, but you do not need to duplicate them as in your example S,E,F. After extracting objects to specific type, you can manipulate them.

  import scala.collection.Map
  import scala.util.parsing.json._

  class CC[T] {
    def unapply(a: Any): Option[T] = Some(a.asInstanceOf[T])
  }

  object M extends CC[Map[String, Any]]
  object L extends CC[List[Any]]
  object S extends CC[String]
  object D extends CC[Double]

  val jsonString2 =
    """
    [{
        "index": 1,
      "source": "a",
      "name": "v",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     },
     {
        "index": 2,
      "source": "b",
      "name": "b",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     }]

    """.stripMargin

  val result = for {
    Some(L(list)) <- List(JSON.parseFull(jsonString2))
    M(map) <- list
    D(index) = map("index")
    S(source) = map("source")
    S(name) = map("name")
    L(s3Connections) = map("s3")
    M(s3) <- s3Connections
    S(path) = s3("path")
    S(bucket) = s3("bucket")
    S(key) = s3("key")
  } yield (index, source, name, path, bucket, key)

  println(result)
  
//  List((1.0,a,v,s3://1,p,r), (2.0,b,b,s3://1,p,r))

Also, you can use any library to parse json files. For instance, I provide below example of working with circe library.

  import io.circe.parser.parse
  import io.circe.{Decoder, Json}
  
  case class S3Config(path: String, bucket: String, key: String)

  object S3Config {
    implicit val decoder: Decoder[S3Config] = deriveDecoder[S3Config]
  }

  case class Source(index: Int, source: String, name: String, s3: Vector[S3Config])

  object Source {
    implicit val decoder: Decoder[Source] = deriveDecoder[Source]
  }

  val jsonString =
    """
    [{
        "index": 1,
      "source": "a",
      "name": "v",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     },
     {
        "index": 2,
      "source": "b",
      "name": "b",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     }]

    """.stripMargin

  val sources = parse(jsonString).getOrElse(Json.Null).as[Vector[Source]].getOrElse(Vector.empty[Source])
  for (source <- sources) {
    println(source.source)
    println(source.index)
    println(source.s3)
  }

  //  a
  //  1
  //  Vector(S3Config(s3://1,p,r))
  //      b
  //  2
  //  Vector(S3Config(s3://1,p,r))

CodePudding user response：

If you want to do it without spark, then you can try this way:

JSON.parseFull(jsonString) match {
    case Some(x) => {
        val json_list = x.asInstanceOf[List[Map[String, Any]]]
        json_list.foreach { json_map =>
            // To extract any base level fields

            // You can do any action on the extracted field. This is just to illustrate the extraction method
            println("index: "   json_map("index"))
            println("source: "   json_map("source"))

            // To extract fields from inside a list: example element at index 0 of list s3

            // You can do any action on the extracted field. This is just to illustrate the extraction method
            val s3_list = json_map("s3").asInstanceOf[List[Any]]
            println("s3: "   s3_list(0))

            // To extract fields from inside a map: example path from 0th element of list s3

            // You can do any action on the extracted field. This is just to illustrate the extraction method
            println("s3.path: "   s3_list(0).asInstanceOf[Map[String, Any]]("path"))

        }
    }
    case None => {
        println("Improper JSON")
    }
}

// Output of the print statements

// Iteration #1:

index: 1.0
source: a
s3: Map(path -> s3://1, bucket -> p, key -> r)
s3.path: s3://1

// Iteration #2:

index: 2.0
source: b
s3: Map(path -> s3://1, bucket -> p, key -> r)
s3.path: s3://1